2023-11-02T08:15:39.062 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-02T08:15:39.064 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888 --verbose --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/orig.config.yaml 2023-11-02T08:15:39.105 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-02T08:15:39.191 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888 branch: wip-yuri6-testing-2023-11-01-0745-reef description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7443888' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri6-testing-2023-11-01-0745-reef ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2023-11-01-0745-reef sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 owner: scheduled_yuriw@teuthology priority: 49 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 525 sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2023-11-01-0745-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 targets: smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKB+YlhA8QRvwhofhT9njZNwkD80E3wr4ljY0MpgRWCrpgX8xVCBUrUlMl/xOz6do5PWtzhVKyqQYgel4+hwS00= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWyM0BTTlZcSU06MdrOc/wX6OC7tSoQIDcXxB4KmJQ+YsjOrG0UckB44ObGExInMq2G8EWSWXFdPBEGaCx14rM= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-01_21:37:41 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-11-02T08:15:39.192 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa; will attempt to use it 2023-11-02T08:15:39.193 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks 2023-11-02T08:15:39.193 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-02T08:15:39.197 INFO:teuthology.task.internal:Checking packages... 2023-11-02T08:15:39.228 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081' 2023-11-02T08:15:39.228 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-02T08:15:39.229 INFO:teuthology.packaging:ref: None 2023-11-02T08:15:39.230 INFO:teuthology.packaging:tag: None 2023-11-02T08:15:39.230 INFO:teuthology.packaging:branch: wip-yuri6-testing-2023-11-01-0745-reef 2023-11-02T08:15:39.230 INFO:teuthology.packaging:sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:15:39.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2023-11-01-0745-reef 2023-11-02T08:15:39.553 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-1159.gb5a9486b 2023-11-02T08:15:39.553 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-02T08:15:39.562 INFO:teuthology.task.internal:no buildpackages task found 2023-11-02T08:15:39.562 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-02T08:15:39.596 INFO:teuthology.task.internal:Saving configuration 2023-11-02T08:15:39.611 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-02T08:15:39.619 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-02T08:15:39.645 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 08:10:14.553469', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKB+YlhA8QRvwhofhT9njZNwkD80E3wr4ljY0MpgRWCrpgX8xVCBUrUlMl/xOz6do5PWtzhVKyqQYgel4+hwS00='} 2023-11-02T08:15:39.669 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 08:10:14.555848', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWyM0BTTlZcSU06MdrOc/wX6OC7tSoQIDcXxB4KmJQ+YsjOrG0UckB44ObGExInMq2G8EWSWXFdPBEGaCx14rM='} 2023-11-02T08:15:39.669 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-02T08:15:39.678 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-11-02T08:15:39.678 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-11-02T08:15:39.678 INFO:teuthology.run_tasks:Running task console_log... 2023-11-02T08:15:39.774 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f64b992ba60>, signals=[15]) 2023-11-02T08:15:39.776 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-02T08:15:39.787 INFO:teuthology.task.internal:Opening connections... 2023-11-02T08:15:39.787 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T08:15:39.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:15:39.876 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T08:15:39.877 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:15:39.959 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-02T08:15:39.968 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2023-11-02T08:15:39.989 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2023-11-02T08:15:39.990 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2023-11-02T08:15:40.046 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2023-11-02T08:15:40.046 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2023-11-02T08:15:40.046 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2023-11-02T08:15:40.047 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2023-11-02T08:15:40.047 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2023-11-02T08:15:40.047 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2023-11-02T08:15:40.047 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-02T08:15:40.047 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2023-11-02T08:15:40.048 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2023-11-02T08:15:40.048 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-02T08:15:40.048 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2023-11-02T08:15:40.048 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-02T08:15:40.048 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-02T08:15:40.049 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-02T08:15:40.049 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2023-11-02T08:15:40.074 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2023-11-02T08:15:40.091 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2023-11-02T08:15:40.091 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2023-11-02T08:15:40.147 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2023-11-02T08:15:40.148 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="9" 2023-11-02T08:15:40.148 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2023-11-02T08:15:40.148 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2023-11-02T08:15:40.148 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="9" 2023-11-02T08:15:40.149 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el9" 2023-11-02T08:15:40.149 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-02T08:15:40.149 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2023-11-02T08:15:40.149 INFO:teuthology.orchestra.run.smithi178.stdout:LOGO="fedora-logo-icon" 2023-11-02T08:15:40.149 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-02T08:15:40.150 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2023-11-02T08:15:40.150 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-02T08:15:40.150 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-02T08:15:40.151 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-02T08:15:40.151 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2023-11-02T08:15:40.174 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-02T08:15:40.183 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-02T08:15:40.204 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-02T08:15:40.204 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2023-11-02T08:15:40.207 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2023-11-02T08:15:40.222 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-02T08:15:40.230 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-02T08:15:40.230 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2023-11-02T08:15:40.263 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2023-11-02T08:15:40.291 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-02T08:15:40.357 INFO:teuthology.run_tasks:Running task kernel... 2023-11-02T08:15:40.377 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-02T08:15:40.378 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-02T08:15:40.378 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-02T08:15:40.379 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-02T08:15:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T08:15:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T08:15:40.396 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2023-11-02T08:15:40.397 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2023-11-02T08:15:40.452 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-361.el9.x86_64 2023-11-02T08:15:40.453 INFO:teuthology.task.kernel:Running kernel on smithi178: 5.14.0-361.el9.x86_64 2023-11-02T08:15:40.453 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2023-11-02T08:15:40.454 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-361.el9.x86_64 2023-11-02T08:15:40.455 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-361.el9.x86_64 2023-11-02T08:15:40.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2023-11-02T08:15:42.733 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - AppStream 12 MB/s | 6.3 MB 00:00 2023-11-02T08:15:42.768 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-11-02T08:15:44.042 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2023-11-02T08:15:44.085 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2023-11-02T08:15:51.300 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 7.9 MB 00:06 2023-11-02T08:16:04.563 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 408 kB/s | 7.9 MB 00:19 2023-11-02T08:16:10.957 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 984 kB/s | 18 MB 00:18 2023-11-02T08:16:16.241 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.8 MB 00:01 2023-11-02T08:16:16.987 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 1.6 MB/s | 18 MB 00:11 2023-11-02T08:16:17.894 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2023-11-02T08:16:19.105 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 18 MB/s | 19 MB 00:01 2023-11-02T08:16:22.465 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 5.8 MB 00:02 2023-11-02T08:16:23.694 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-11-02T08:16:23.871 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2023-11-02T08:16:25.775 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-02T08:16:25.775 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-02T08:16:25.873 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:16:25.888 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:16:25.889 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2023-11-02T08:16:25.889 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:16:25.889 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:16:25.889 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-11-02T08:16:25.889 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:16:25.890 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-11-02T08:16:25.890 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-11-02T08:16:25.890 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-11-02T08:16:25.890 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:16:25.890 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:16:25.891 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:16:25.891 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2023-11-02T08:16:25.891 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:16:25.892 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 95 M 2023-11-02T08:16:25.892 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 120 M 2023-11-02T08:16:25.892 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:16:26.870 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 6.8 MB/s | 5.3 MB 00:00 2023-11-02T08:16:27.437 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 15 MB/s | 20 MB 00:01 2023-11-02T08:16:28.128 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 18 MB/s | 38 MB 00:02 2023-11-02T08:16:28.283 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 19 MB 00:04 2023-11-02T08:16:28.854 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 16 MB/s | 32 MB 00:01 2023-11-02T08:16:28.854 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:16:28.854 INFO:teuthology.orchestra.run.smithi151.stdout:Total 32 MB/s | 95 MB 00:02 2023-11-02T08:16:29.697 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:16:30.071 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:16:30.071 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:16:31.372 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:16:31.373 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:16:32.830 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-11-02T08:16:33.996 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:16:34.629 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-11-02T08:16:34.779 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-02T08:16:34.912 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-02T08:16:34.912 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-02T08:16:35.005 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:16:35.014 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:16:35.015 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-02T08:16:35.015 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:16:35.015 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:16:35.015 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-11-02T08:16:35.016 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:16:35.016 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-11-02T08:16:35.016 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-11-02T08:16:35.016 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-11-02T08:16:35.016 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:16:35.017 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:16:35.017 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:16:35.017 INFO:teuthology.orchestra.run.smithi178.stdout:Install 4 Packages 2023-11-02T08:16:35.017 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:16:35.017 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 95 M 2023-11-02T08:16:35.018 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 120 M 2023-11-02T08:16:35.018 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:16:37.558 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-02T08:16:37.740 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-02T08:16:37.898 INFO:teuthology.orchestra.run.smithi178.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 2.0 MB/s | 5.3 MB 00:02 2023-11-02T08:16:38.823 INFO:teuthology.orchestra.run.smithi178.stdout:(2/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 10 MB/s | 38 MB 00:03 2023-11-02T08:16:40.531 INFO:teuthology.orchestra.run.smithi178.stdout:(3/4): kernel-core-5.14.0-378.el9.x86_64.rpm 3.8 MB/s | 20 MB 00:05 2023-11-02T08:16:41.464 INFO:teuthology.orchestra.run.smithi178.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 9.0 MB/s | 32 MB 00:03 2023-11-02T08:16:41.465 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:16:41.465 INFO:teuthology.orchestra.run.smithi178.stdout:Total 15 MB/s | 95 MB 00:06 2023-11-02T08:16:42.312 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:16:42.331 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-02T08:16:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:16:42.682 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:16:42.682 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:16:43.964 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:16:43.964 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:16:46.561 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:16:47.078 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:16:47.195 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-11-02T08:16:47.322 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-02T08:16:50.101 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-02T08:16:50.300 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-02T08:16:54.911 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-02T08:16:55.115 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:16:59.784 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:17:16.341 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:17:16.383 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:17:16.870 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:17:16.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-11-02T08:17:16.871 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-02T08:17:16.871 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-02T08:17:17.292 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:17:17.293 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:17:17.293 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:17:17.293 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-378.el9.x86_64 2023-11-02T08:17:17.293 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-11-02T08:17:17.294 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-11-02T08:17:17.294 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-11-02T08:17:17.294 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:17:17.294 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:17:17.839 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2023-11-02T08:17:18.022 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-02T08:17:18.023 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-02T08:17:18.023 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-11-02T08:17:18.023 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-02T08:17:18.024 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-02T08:17:18.024 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-02T08:17:18.024 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2023-11-02T08:17:18.024 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2023-11-02T08:17:18.141 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-02T08:17:18.142 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-02T08:17:18.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2023-11-02T08:17:18.184 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2023-11-02T08:17:18.185 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2023-11-02T08:17:18.185 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2023-11-02T08:17:18.185 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 68.el9 2023-11-02T08:17:18.185 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2023-11-02T08:17:18.185 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-02T08:17:18.186 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2023-11-02T08:17:18.186 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8264254 2023-11-02T08:17:18.186 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2023-11-02T08:17:18.186 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-02T08:17:18.187 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-02T08:17:18.187 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-02T08:17:18.187 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-02T08:17:18.187 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2023-11-02T08:17:18.187 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2023-11-02T08:17:18.188 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-02T08:17:18.188 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2023-11-02T08:17:18.188 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2023-11-02T08:17:18.188 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:17:18.188 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-02T08:17:18.188 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-02T08:17:18.189 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-02T08:17:18.189 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2023-11-02T08:17:18.189 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:17:18.190 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2023-11-02T08:17:18.191 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-02T08:17:18.191 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-378.el9.x86_64 2023-11-02T08:17:18.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-02T08:17:18.825 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2023-11-02T08:17:20.252 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-02T08:17:20.293 INFO:teuthology.orchestra.run.smithi151.stderr:done 2023-11-02T08:17:20.299 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2023-11-02T08:17:20.314 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.LqEf9r692o 2023-11-02T08:17:20.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LqEf9r692o 2023-11-02T08:17:20.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.LqEf9r692o 2023-11-02T08:17:20.525 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.LqEf9r692o is 6KB 2023-11-02T08:17:20.574 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.LqEf9r692o 2023-11-02T08:17:20.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2023-11-02T08:17:20.655 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-02T08:17:20.655 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-02T08:17:20.655 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-02T08:17:20.656 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-11-02T08:17:20.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-11-02T08:17:20.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2023-11-02T08:17:28.373 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:17:28.415 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:17:28.806 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:17:28.806 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-11-02T08:17:28.807 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-02T08:17:28.807 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-02T08:17:29.247 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:17:29.247 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:17:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:17:29.248 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-5.14.0-378.el9.x86_64 2023-11-02T08:17:29.248 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-11-02T08:17:29.248 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-11-02T08:17:29.248 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-11-02T08:17:29.248 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:17:29.249 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:17:29.740 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2023-11-02T08:17:29.927 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-02T08:17:29.927 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-02T08:17:29.928 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-11-02T08:17:29.928 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-02T08:17:29.928 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-02T08:17:29.928 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-11-02T08:17:29.929 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2023-11-02T08:17:29.929 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2023-11-02T08:17:30.045 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-02T08:17:30.046 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-02T08:17:30.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -qi grub2-tools 2023-11-02T08:17:30.081 INFO:teuthology.orchestra.run.smithi178.stdout:Name : grub2-tools 2023-11-02T08:17:30.081 INFO:teuthology.orchestra.run.smithi178.stdout:Epoch : 1 2023-11-02T08:17:30.081 INFO:teuthology.orchestra.run.smithi178.stdout:Version : 2.06 2023-11-02T08:17:30.081 INFO:teuthology.orchestra.run.smithi178.stdout:Release : 68.el9 2023-11-02T08:17:30.082 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: x86_64 2023-11-02T08:17:30.082 INFO:teuthology.orchestra.run.smithi178.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-02T08:17:30.082 INFO:teuthology.orchestra.run.smithi178.stdout:Group : Unspecified 2023-11-02T08:17:30.082 INFO:teuthology.orchestra.run.smithi178.stdout:Size : 8264254 2023-11-02T08:17:30.082 INFO:teuthology.orchestra.run.smithi178.stdout:License : GPLv3+ 2023-11-02T08:17:30.083 INFO:teuthology.orchestra.run.smithi178.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-02T08:17:30.083 INFO:teuthology.orchestra.run.smithi178.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-02T08:17:30.083 INFO:teuthology.orchestra.run.smithi178.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-02T08:17:30.083 INFO:teuthology.orchestra.run.smithi178.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-02T08:17:30.083 INFO:teuthology.orchestra.run.smithi178.stdout:Packager : builder@centos.org 2023-11-02T08:17:30.084 INFO:teuthology.orchestra.run.smithi178.stdout:Vendor : CentOS 2023-11-02T08:17:30.084 INFO:teuthology.orchestra.run.smithi178.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-02T08:17:30.084 INFO:teuthology.orchestra.run.smithi178.stdout:Summary : Support tools for GRUB. 2023-11-02T08:17:30.084 INFO:teuthology.orchestra.run.smithi178.stdout:Description : 2023-11-02T08:17:30.084 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:17:30.085 INFO:teuthology.orchestra.run.smithi178.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-02T08:17:30.085 INFO:teuthology.orchestra.run.smithi178.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-02T08:17:30.085 INFO:teuthology.orchestra.run.smithi178.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-02T08:17:30.085 INFO:teuthology.orchestra.run.smithi178.stdout:hardware devices. 2023-11-02T08:17:30.085 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:17:30.086 INFO:teuthology.orchestra.run.smithi178.stdout:This subpackage provides tools for support of all platforms. 2023-11-02T08:17:30.087 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-02T08:17:30.087 INFO:teuthology.task.kernel:Updating grub on smithi178 to boot 5.14.0-378.el9.x86_64 2023-11-02T08:17:30.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-02T08:17:30.688 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2023-11-02T08:17:32.043 INFO:teuthology.orchestra.run.smithi178.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-02T08:17:32.078 INFO:teuthology.orchestra.run.smithi178.stderr:done 2023-11-02T08:17:32.081 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2023-11-02T08:17:32.097 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.zoBW7ua6uk 2023-11-02T08:17:32.097 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zoBW7ua6uk 2023-11-02T08:17:32.167 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.zoBW7ua6uk 2023-11-02T08:17:32.309 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.zoBW7ua6uk is 6KB 2023-11-02T08:17:32.359 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.zoBW7ua6uk 2023-11-02T08:17:32.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo /bin/ls /boot/loader/entries || true 2023-11-02T08:17:32.448 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-02T08:17:32.448 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-02T08:17:32.448 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-02T08:17:32.448 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-11-02T08:17:32.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-11-02T08:17:32.612 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2023-11-02T08:17:50.817 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-02T08:17:50.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-11-02T08:17:50.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:18:02.627 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-11-02T08:18:02.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-02T08:18:02.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:18:25.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2023-11-02T08:18:34.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-11-02T08:18:34.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:18:37.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-11-02T08:18:46.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-02T08:18:46.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:18:49.370 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-11-02T08:19:01.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-02T08:19:01.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:19:33.195 DEBUG:teuthology.orchestra.run.smithi178:> true 2023-11-02T08:19:33.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-02T08:19:33.711 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-378.el9.x86_64"... 2023-11-02T08:19:33.711 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2023-11-02T08:19:33.726 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-378.el9.x86_64 2023-11-02T08:19:33.728 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-11-02T08:19:33.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-02T08:19:33.729 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-02T08:19:34.288 DEBUG:teuthology.orchestra.remote:timed out 2023-11-02T08:19:34.730 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-02T08:19:34.731 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-02T08:19:34.805 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2023-11-02T08:19:34.867 DEBUG:teuthology.parallel:result is None 2023-11-02T08:19:46.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-11-02T08:19:46.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:19:46.791 DEBUG:teuthology.orchestra.run.smithi151:> true 2023-11-02T08:19:47.148 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-11-02T08:19:47.149 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-11-02T08:19:47.149 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2023-11-02T08:19:47.164 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-378.el9.x86_64 2023-11-02T08:19:47.165 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-11-02T08:19:47.165 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-02T08:19:47.165 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-02T08:19:48.166 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-02T08:19:48.167 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-02T08:19:48.244 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2023-11-02T08:19:48.281 DEBUG:teuthology.parallel:result is None 2023-11-02T08:19:48.281 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-02T08:19:48.289 INFO:teuthology.task.internal:Creating test directory... 2023-11-02T08:19:48.290 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-02T08:19:48.293 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-02T08:19:48.311 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-02T08:19:48.332 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-02T08:19:48.382 INFO:teuthology.task.internal:Creating archive directory... 2023-11-02T08:19:48.382 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-02T08:19:48.385 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-02T08:19:48.436 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-02T08:19:48.444 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-02T08:19:48.444 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-02T08:19:48.471 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-02T08:19:48.506 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:19:48.509 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:19:48.523 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:19:48.523 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:19:48.527 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-02T08:19:48.535 INFO:teuthology.task.internal:Configuring sudo... 2023-11-02T08:19:48.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-02T08:19:48.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-02T08:19:48.600 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-02T08:19:48.610 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-02T08:19:48.611 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-02T08:19:48.640 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-02T08:19:48.658 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-02T08:19:48.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-02T08:19:48.800 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-02T08:19:48.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-02T08:19:48.927 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:19:48.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-02T08:19:48.994 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-02T08:19:49.042 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-02T08:19:49.100 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-02T08:19:49.185 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-02T08:19:49.229 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:19:49.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-02T08:19:49.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2023-11-02T08:19:49.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2023-11-02T08:19:49.340 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-02T08:19:49.381 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-02T08:19:49.813 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-02T08:19:49.821 INFO:teuthology.task.internal:Starting timer... 2023-11-02T08:19:49.821 INFO:teuthology.run_tasks:Running task pcp... 2023-11-02T08:19:49.860 INFO:teuthology.run_tasks:Running task selinux... 2023-11-02T08:19:49.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2023-11-02T08:19:49.970 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2023-11-02T08:19:49.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2023-11-02T08:19:50.019 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2023-11-02T08:19:50.020 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-02T08:19:50.021 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2023-11-02T08:19:50.040 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2023-11-02T08:19:50.041 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2023-11-02T08:19:50.066 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2023-11-02T08:19:50.066 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi151.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2023-11-02T08:19:50.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-02T08:19:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T08:19:50.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-02T08:19:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T08:19:50.141 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-02T08:19:50.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2023-11-02T08:19:50.182 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2023-11-02T08:19:50.218 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-02T08:19:50.228 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-02T08:19:50.228 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-02T08:19:50.247 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-02T08:19:50.249 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi151.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-02T08:25:31.026 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi151.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2023-11-02T08:25:31.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-11-02T08:25:31.029 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:25:31.112 DEBUG:teuthology.orchestra.run.smithi151:> true 2023-11-02T08:25:31.184 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-11-02T08:25:31.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-02T08:25:31.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:25:31.268 DEBUG:teuthology.orchestra.run.smithi178:> true 2023-11-02T08:25:31.342 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-02T08:25:31.342 INFO:teuthology.run_tasks:Running task clock... 2023-11-02T08:25:31.358 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-02T08:25:31.359 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-02T08:25:31.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T08:25:31.362 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-02T08:25:31.362 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T08:25:31.390 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-02T08:25:31.406 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-02T08:25:31.414 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-02T08:25:31.431 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-02T08:25:31.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2023-11-02T08:25:31.449 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2023-11-02T08:25:31.460 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2023-11-02T08:25:31.465 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-02T08:25:31.474 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2023-11-02T08:25:31.482 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-02T08:25:31.490 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-02T08:25:31.506 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-02T08:25:31.525 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2023-11-02T08:25:31.528 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-02T08:25:31.528 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2023-11-02T08:25:31.528 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:25:31.528 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:25:31.528 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:25:31.529 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:25:31.549 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2023-11-02T08:25:31.552 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-02T08:25:31.552 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2023-11-02T08:25:31.553 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:25:31.553 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:25:31.553 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:25:31.553 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:25:31.554 INFO:teuthology.run_tasks:Running task install... 2023-11-02T08:25:31.563 DEBUG:teuthology.task.install:project ceph 2023-11-02T08:25:31.564 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081'}} 2023-11-02T08:25:31.564 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081'} 2023-11-02T08:25:31.564 INFO:teuthology.task.install:Using flavor: default 2023-11-02T08:25:31.570 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-11-02T08:25:31.571 INFO:teuthology.task.install:extra packages: [] 2023-11-02T08:25:31.571 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081', 'tag': None, 'wait_for_package': False} 2023-11-02T08:25:31.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:25:31.574 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081', 'tag': None, 'wait_for_package': False} 2023-11-02T08:25:31.574 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:25:31.719 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/ 2023-11-02T08:25:31.720 INFO:teuthology.task.install.rpm:Package version is 18.2.0-1159.gb5a9486b 2023-11-02T08:25:31.727 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/ 2023-11-02T08:25:31.727 INFO:teuthology.task.install.rpm:Package version is 18.2.0-1159.gb5a9486b 2023-11-02T08:25:31.846 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-02T08:25:31.847 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:25:31.848 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-02T08:25:31.851 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-02T08:25:31.852 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:25:31.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-02T08:25:31.879 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-11-02T08:25:31.879 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-02T08:25:31.880 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-11-02T08:25:31.881 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-02T08:25:31.950 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-02T08:25:31.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-02T08:25:32.030 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-02T08:25:32.032 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-02T08:25:32.097 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2023-11-02T08:25:32.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2023-11-02T08:25:32.099 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2023-11-02T08:25:32.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2023-11-02T08:25:32.352 INFO:teuthology.orchestra.run.smithi151.stdout:72 files removed 2023-11-02T08:25:32.356 INFO:teuthology.orchestra.run.smithi178.stdout:72 files removed 2023-11-02T08:25:32.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw 2023-11-02T08:25:32.387 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw 2023-11-02T08:25:33.054 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 219 kB/s | 77 kB 00:00 2023-11-02T08:25:33.061 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 210 kB/s | 77 kB 00:00 2023-11-02T08:25:33.405 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 31 kB/s | 9.4 kB 00:00 2023-11-02T08:25:33.428 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 30 kB/s | 9.4 kB 00:00 2023-11-02T08:25:33.771 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2023-11-02T08:25:33.772 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2023-11-02T08:25:34.148 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 18 MB/s | 6.3 MB 00:00 2023-11-02T08:25:34.158 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - AppStream 18 MB/s | 6.3 MB 00:00 2023-11-02T08:25:35.465 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2023-11-02T08:25:35.742 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2023-11-02T08:25:38.858 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 7.9 MB 00:02 2023-11-02T08:25:39.689 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 7.9 MB 00:03 2023-11-02T08:25:41.776 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 17 MB/s | 18 MB 00:01 2023-11-02T08:25:45.708 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 5.8 MB 00:00 2023-11-02T08:25:46.962 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 18 MB 00:07 2023-11-02T08:25:47.242 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 15 kB 00:00 2023-11-02T08:25:48.655 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 15 MB/s | 19 MB 00:01 2023-11-02T08:25:52.484 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 5.8 MB 00:02 2023-11-02T08:25:53.177 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2023-11-02T08:25:54.293 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2023-11-02T08:25:55.565 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:25:55.576 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:25:55.576 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:25:55.576 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:25:55.576 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:25:55.577 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 7.7 M 2023-11-02T08:25:55.577 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2023-11-02T08:25:55.577 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 3.3 M 2023-11-02T08:25:55.577 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 3.0 M 2023-11-02T08:25:55.577 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:25:55.578 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options 2023-11-02T08:25:55.578 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-02T08:25:55.578 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 5.2 M 2023-11-02T08:25:55.578 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 18 M 2023-11-02T08:25:55.579 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 24 k 2023-11-02T08:25:55.579 INFO:teuthology.orchestra.run.smithi151.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-11-02T08:25:55.579 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-02T08:25:55.579 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-02T08:25:55.579 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-02T08:25:55.580 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 659 k 2023-11-02T08:25:55.580 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-02T08:25:55.580 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-02T08:25:55.580 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-11-02T08:25:55.580 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 475 k 2023-11-02T08:25:55.581 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-02T08:25:55.581 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 4.4 M 2023-11-02T08:25:55.581 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-02T08:25:55.581 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-02T08:25:55.582 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-02T08:25:55.582 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse 2023-11-02T08:25:55.582 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 45 k 2023-11-02T08:25:55.582 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 124 k 2023-11-02T08:25:55.583 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 162 k 2023-11-02T08:25:55.583 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-02T08:25:55.583 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 320 k 2023-11-02T08:25:55.583 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 298 k 2023-11-02T08:25:55.583 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 99 k 2023-11-02T08:25:55.584 INFO:teuthology.orchestra.run.smithi151.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-02T08:25:55.584 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-02T08:25:55.584 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:25:55.584 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:25:55.585 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:25:55.585 INFO:teuthology.orchestra.run.smithi151.stdout:Install 28 Packages 2023-11-02T08:25:55.585 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2023-11-02T08:25:55.585 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:25:55.586 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 53 M 2023-11-02T08:25:55.586 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:25:57.011 INFO:teuthology.orchestra.run.smithi151.stdout:(1/30): ceph-base-18.2.0-1159.gb5a9486b.el9.x86 6.4 MB/s | 5.2 MB 00:00 2023-11-02T08:25:57.070 INFO:teuthology.orchestra.run.smithi151.stdout:(2/30): ceph-selinux-18.2.0-1159.gb5a9486b.el9. 414 kB/s | 24 kB 00:00 2023-11-02T08:25:57.203 INFO:teuthology.orchestra.run.smithi151.stdout:(3/30): ceph-radosgw-18.2.0-1159.gb5a9486b.el9. 7.7 MB/s | 7.7 MB 00:00 2023-11-02T08:25:57.237 INFO:teuthology.orchestra.run.smithi151.stdout:(4/30): libcephfs2-18.2.0-1159.gb5a9486b.el9.x8 3.9 MB/s | 659 kB 00:00 2023-11-02T08:25:57.270 INFO:teuthology.orchestra.run.smithi151.stdout:(5/30): libradosstriper1-18.2.0-1159.gb5a9486b. 7.0 MB/s | 475 kB 00:00 2023-11-02T08:25:57.337 INFO:teuthology.orchestra.run.smithi151.stdout:(6/30): python3-ceph-argparse-18.2.0-1159.gb5a9 676 kB/s | 45 kB 00:00 2023-11-02T08:25:57.413 INFO:teuthology.orchestra.run.smithi151.stdout:(7/30): python3-ceph-common-18.2.0-1159.gb5a948 1.6 MB/s | 124 kB 00:00 2023-11-02T08:25:57.488 INFO:teuthology.orchestra.run.smithi151.stdout:(8/30): python3-cephfs-18.2.0-1159.gb5a9486b.el 2.1 MB/s | 162 kB 00:00 2023-11-02T08:25:57.572 INFO:teuthology.orchestra.run.smithi151.stdout:(9/30): python3-rados-18.2.0-1159.gb5a9486b.el9 3.8 MB/s | 320 kB 00:00 2023-11-02T08:25:57.647 INFO:teuthology.orchestra.run.smithi151.stdout:(10/30): librgw2-18.2.0-1159.gb5a9486b.el9.x86_ 11 MB/s | 4.4 MB 00:00 2023-11-02T08:25:57.672 INFO:teuthology.orchestra.run.smithi151.stdout:(11/30): python3-rbd-18.2.0-1159.gb5a9486b.el9. 2.9 MB/s | 298 kB 00:00 2023-11-02T08:25:57.731 INFO:teuthology.orchestra.run.smithi151.stdout:(12/30): python3-rgw-18.2.0-1159.gb5a9486b.el9. 1.2 MB/s | 99 kB 00:00 2023-11-02T08:25:57.781 INFO:teuthology.orchestra.run.smithi151.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 974 kB/s | 106 kB 00:00 2023-11-02T08:25:57.806 INFO:teuthology.orchestra.run.smithi151.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2023-11-02T08:25:57.849 INFO:teuthology.orchestra.run.smithi151.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-11-02T08:25:57.882 INFO:teuthology.orchestra.run.smithi151.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-11-02T08:25:57.899 INFO:teuthology.orchestra.run.smithi151.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2023-11-02T08:25:58.091 INFO:teuthology.orchestra.run.smithi151.stdout:(18/30): ceph-common-18.2.0-1159.gb5a9486b.el9. 9.7 MB/s | 18 MB 00:01 2023-11-02T08:25:58.175 INFO:teuthology.orchestra.run.smithi151.stdout:(19/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 155 kB/s | 45 kB 00:00 2023-11-02T08:25:58.292 INFO:teuthology.orchestra.run.smithi151.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 282 kB/s | 111 kB 00:00 2023-11-02T08:25:58.333 INFO:teuthology.orchestra.run.smithi151.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 585 kB/s | 24 kB 00:00 2023-11-02T08:25:58.367 INFO:teuthology.orchestra.run.smithi151.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2023-11-02T08:25:58.392 INFO:teuthology.orchestra.run.smithi151.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 828 kB/s | 49 kB 00:00 2023-11-02T08:25:58.417 INFO:teuthology.orchestra.run.smithi151.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2023-11-02T08:25:58.500 INFO:teuthology.orchestra.run.smithi151.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2023-11-02T08:25:58.684 INFO:teuthology.orchestra.run.smithi151.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.8 MB/s | 837 kB 00:00 2023-11-02T08:25:58.719 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 19 MB 00:04 2023-11-02T08:25:58.934 INFO:teuthology.orchestra.run.smithi151.stdout:(27/30): librados2-18.2.0-1159.gb5a9486b.el9.x8 13 MB/s | 3.3 MB 00:00 2023-11-02T08:25:59.110 INFO:teuthology.orchestra.run.smithi151.stdout:(28/30): librbd1-18.2.0-1159.gb5a9486b.el9.x86_ 17 MB/s | 3.0 MB 00:00 2023-11-02T08:25:59.152 INFO:teuthology.orchestra.run.smithi151.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 2.4 MB/s | 1.6 MB 00:00 2023-11-02T08:26:00.085 INFO:teuthology.orchestra.run.smithi151.stdout:(30/30): libarrow-9.0.0-9.el9.x86_64.rpm 2.3 MB/s | 4.4 MB 00:01 2023-11-02T08:26:00.092 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:00.092 INFO:teuthology.orchestra.run.smithi151.stdout:Total 12 MB/s | 53 MB 00:04 2023-11-02T08:26:00.105 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:26:00.143 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:26:00.143 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:26:00.627 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:26:00.628 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:26:02.045 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:26:02.106 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-11-02T08:26:02.154 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-11-02T08:26:02.216 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-11-02T08:26:02.527 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-11-02T08:26:02.589 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/32 2023-11-02T08:26:02.663 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/32 2023-11-02T08:26:02.745 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 6/32 2023-11-02T08:26:02.782 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 7/32 2023-11-02T08:26:02.828 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 7/32 2023-11-02T08:26:02.857 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-11-02T08:26:02.901 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:18.2.0-1159.gb5a9486b.el9. 9/32 2023-11-02T08:26:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 10/32 2023-11-02T08:26:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_6 11/32 2023-11-02T08:26:03.058 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_6 11/32 2023-11-02T08:26:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-11-02T08:26:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-11-02T08:26:03.234 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-11-02T08:26:03.235 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2023-11-02T08:26:03.266 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-11-02T08:26:03.293 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-11-02T08:26:03.330 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-11-02T08:26:03.371 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-02T08:26:03.688 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-11-02T08:26:03.715 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 20/32 2023-11-02T08:26:03.788 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 20/32 2023-11-02T08:26:03.828 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 21/32 2023-11-02T08:26:03.864 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-11-02T08:26:04.265 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-11-02T08:26:04.293 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 24/32 2023-11-02T08:26:04.364 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 24/32 2023-11-02T08:26:04.432 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 25/32 2023-11-02T08:26:04.471 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:18.2.0-1159.gb5a9486b.el9.x8 26/32 2023-11-02T08:26:05.590 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:26:05.594 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:05.594 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:26:05.594 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:05.594 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:26:05.594 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 7.7 M 2023-11-02T08:26:05.595 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2023-11-02T08:26:05.595 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 3.3 M 2023-11-02T08:26:05.595 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 3.0 M 2023-11-02T08:26:05.595 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:26:05.596 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options 2023-11-02T08:26:05.596 INFO:teuthology.orchestra.run.smithi178.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-02T08:26:05.596 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 5.2 M 2023-11-02T08:26:05.596 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 18 M 2023-11-02T08:26:05.596 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 24 k 2023-11-02T08:26:05.597 INFO:teuthology.orchestra.run.smithi178.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-11-02T08:26:05.597 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-02T08:26:05.597 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-02T08:26:05.597 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-02T08:26:05.597 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 659 k 2023-11-02T08:26:05.598 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-02T08:26:05.598 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-02T08:26:05.598 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-11-02T08:26:05.598 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 475 k 2023-11-02T08:26:05.598 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-02T08:26:05.599 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 4.4 M 2023-11-02T08:26:05.599 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-02T08:26:05.599 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-02T08:26:05.599 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-02T08:26:05.599 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse 2023-11-02T08:26:05.600 INFO:teuthology.orchestra.run.smithi178.stdout: x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 45 k 2023-11-02T08:26:05.600 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 124 k 2023-11-02T08:26:05.600 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 162 k 2023-11-02T08:26:05.600 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-02T08:26:05.600 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 320 k 2023-11-02T08:26:05.601 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 298 k 2023-11-02T08:26:05.601 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 99 k 2023-11-02T08:26:05.601 INFO:teuthology.orchestra.run.smithi178.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-02T08:26:05.601 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-02T08:26:05.601 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:05.602 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:26:05.602 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:05.602 INFO:teuthology.orchestra.run.smithi178.stdout:Install 28 Packages 2023-11-02T08:26:05.602 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 2 Packages 2023-11-02T08:26:05.603 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:05.603 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 53 M 2023-11-02T08:26:05.604 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:26:06.424 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 27/32 2023-11-02T08:26:06.461 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 27/32 2023-11-02T08:26:07.036 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 27/32 2023-11-02T08:26:07.074 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 28/32 2023-11-02T08:26:07.143 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 28/32 2023-11-02T08:26:07.144 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-02T08:26:07.144 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-02T08:26:07.144 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:07.175 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/32 2023-11-02T08:26:07.362 INFO:teuthology.orchestra.run.smithi178.stdout:(1/30): ceph-base-18.2.0-1159.gb5a9486b.el9.x86 4.6 MB/s | 5.2 MB 00:01 2023-11-02T08:26:07.504 INFO:teuthology.orchestra.run.smithi178.stdout:(2/30): ceph-radosgw-18.2.0-1159.gb5a9486b.el9. 6.0 MB/s | 7.7 MB 00:01 2023-11-02T08:26:07.529 INFO:teuthology.orchestra.run.smithi178.stdout:(3/30): ceph-selinux-18.2.0-1159.gb5a9486b.el9. 144 kB/s | 24 kB 00:00 2023-11-02T08:26:07.629 INFO:teuthology.orchestra.run.smithi178.stdout:(4/30): libcephfs2-18.2.0-1159.gb5a9486b.el9.x8 5.1 MB/s | 659 kB 00:00 2023-11-02T08:26:07.654 INFO:teuthology.orchestra.run.smithi178.stdout:(5/30): libradosstriper1-18.2.0-1159.gb5a9486b. 3.7 MB/s | 475 kB 00:00 2023-11-02T08:26:07.730 INFO:teuthology.orchestra.run.smithi178.stdout:(6/30): python3-ceph-argparse-18.2.0-1159.gb5a9 600 kB/s | 45 kB 00:00 2023-11-02T08:26:07.805 INFO:teuthology.orchestra.run.smithi178.stdout:(7/30): python3-ceph-common-18.2.0-1159.gb5a948 1.6 MB/s | 124 kB 00:00 2023-11-02T08:26:07.897 INFO:teuthology.orchestra.run.smithi178.stdout:(8/30): python3-cephfs-18.2.0-1159.gb5a9486b.el 1.7 MB/s | 162 kB 00:00 2023-11-02T08:26:07.997 INFO:teuthology.orchestra.run.smithi178.stdout:(9/30): python3-rados-18.2.0-1159.gb5a9486b.el9 3.1 MB/s | 320 kB 00:00 2023-11-02T08:26:08.279 INFO:teuthology.orchestra.run.smithi178.stdout:(10/30): ceph-common-18.2.0-1159.gb5a9486b.el9. 8.9 MB/s | 18 MB 00:02 2023-11-02T08:26:08.307 INFO:teuthology.orchestra.run.smithi178.stdout:(11/30): python3-rbd-18.2.0-1159.gb5a9486b.el9. 965 kB/s | 298 kB 00:00 2023-11-02T08:26:08.332 INFO:teuthology.orchestra.run.smithi178.stdout:(12/30): python3-rgw-18.2.0-1159.gb5a9486b.el9. 1.9 MB/s | 99 kB 00:00 2023-11-02T08:26:08.399 INFO:teuthology.orchestra.run.smithi178.stdout:(13/30): librgw2-18.2.0-1159.gb5a9486b.el9.x86_ 5.8 MB/s | 4.4 MB 00:00 2023-11-02T08:26:08.524 INFO:teuthology.orchestra.run.smithi178.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 487 kB/s | 106 kB 00:00 2023-11-02T08:26:08.549 INFO:teuthology.orchestra.run.smithi178.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 732 kB/s | 159 kB 00:00 2023-11-02T08:26:08.574 INFO:teuthology.orchestra.run.smithi178.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2023-11-02T08:26:08.625 INFO:teuthology.orchestra.run.smithi178.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 551 kB/s | 41 kB 00:00 2023-11-02T08:26:08.650 INFO:teuthology.orchestra.run.smithi178.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 661 kB 00:00 2023-11-02T08:26:08.709 INFO:teuthology.orchestra.run.smithi178.stdout:(19/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 338 kB/s | 45 kB 00:00 2023-11-02T08:26:09.209 INFO:teuthology.orchestra.run.smithi178.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 190 kB/s | 111 kB 00:00 2023-11-02T08:26:09.352 INFO:teuthology.orchestra.run.smithi178.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 441 kB/s | 309 kB 00:00 2023-11-02T08:26:09.377 INFO:teuthology.orchestra.run.smithi178.stdout:(22/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 146 kB/s | 24 kB 00:00 2023-11-02T08:26:09.502 INFO:teuthology.orchestra.run.smithi178.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 323 kB/s | 49 kB 00:00 2023-11-02T08:26:09.985 INFO:teuthology.orchestra.run.smithi178.stdout:(24/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.7 MB/s | 837 kB 00:00 2023-11-02T08:26:10.144 INFO:teuthology.orchestra.run.smithi178.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2023-11-02T08:26:10.794 INFO:teuthology.orchestra.run.smithi178.stdout:(26/30): thrift-0.15.0-2.el9.x86_64.rpm 2.4 MB/s | 1.6 MB 00:00 2023-11-02T08:26:11.061 INFO:teuthology.orchestra.run.smithi178.stdout:(27/30): libunwind-1.6.2-1.el9.x86_64.rpm 40 kB/s | 67 kB 00:01 2023-11-02T08:26:11.170 INFO:teuthology.orchestra.run.smithi178.stdout:(28/30): librados2-18.2.0-1159.gb5a9486b.el9.x8 8.7 MB/s | 3.3 MB 00:00 2023-11-02T08:26:11.270 INFO:teuthology.orchestra.run.smithi178.stdout:(29/30): libarrow-9.0.0-9.el9.x86_64.rpm 1.7 MB/s | 4.4 MB 00:02 2023-11-02T08:26:11.403 INFO:teuthology.orchestra.run.smithi178.stdout:(30/30): librbd1-18.2.0-1159.gb5a9486b.el9.x86_ 8.8 MB/s | 3.0 MB 00:00 2023-11-02T08:26:11.409 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:11.409 INFO:teuthology.orchestra.run.smithi178.stdout:Total 9.1 MB/s | 53 MB 00:05 2023-11-02T08:26:11.423 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:26:11.461 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:26:11.461 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:26:11.936 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:26:11.937 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:26:13.355 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:26:13.424 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-11-02T08:26:13.511 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-11-02T08:26:13.576 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-11-02T08:26:13.882 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-11-02T08:26:13.941 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/32 2023-11-02T08:26:14.026 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/32 2023-11-02T08:26:14.113 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 6/32 2023-11-02T08:26:14.151 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 7/32 2023-11-02T08:26:14.220 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 7/32 2023-11-02T08:26:14.250 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-11-02T08:26:14.295 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:18.2.0-1159.gb5a9486b.el9. 9/32 2023-11-02T08:26:14.358 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 10/32 2023-11-02T08:26:14.385 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_6 11/32 2023-11-02T08:26:14.452 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_6 11/32 2023-11-02T08:26:14.557 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-11-02T08:26:14.594 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-11-02T08:26:14.637 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-11-02T08:26:14.667 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-11-02T08:26:14.685 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-11-02T08:26:14.713 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-11-02T08:26:14.745 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-02T08:26:15.042 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-11-02T08:26:15.075 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 20/32 2023-11-02T08:26:15.183 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 20/32 2023-11-02T08:26:15.213 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 21/32 2023-11-02T08:26:15.250 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-11-02T08:26:15.646 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-11-02T08:26:15.678 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 24/32 2023-11-02T08:26:15.749 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 24/32 2023-11-02T08:26:15.824 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 25/32 2023-11-02T08:26:15.856 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:18.2.0-1159.gb5a9486b.el9.x8 26/32 2023-11-02T08:26:16.891 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/32 2023-11-02T08:26:16.892 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2023-11-02T08:26:16.892 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2023-11-02T08:26:16.892 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2023-11-02T08:26:16.892 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2023-11-02T08:26:16.892 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2023-11-02T08:26:16.893 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2023-11-02T08:26:16.893 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2023-11-02T08:26:16.893 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:17.634 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/32 2023-11-02T08:26:17.659 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/32 2023-11-02T08:26:17.659 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:17.660 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-02T08:26:17.660 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-02T08:26:17.661 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-02T08:26:17.661 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:17.662 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-02T08:26:17.753 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-02T08:26:17.753 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-02T08:26:17.857 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 27/32 2023-11-02T08:26:17.908 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 27/32 2023-11-02T08:26:18.458 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 27/32 2023-11-02T08:26:18.495 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 28/32 2023-11-02T08:26:18.586 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 28/32 2023-11-02T08:26:18.587 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-02T08:26:18.587 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-02T08:26:18.587 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:18.621 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/32 2023-11-02T08:26:18.840 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-02T08:26:18.840 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/32 2023-11-02T08:26:18.840 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/32 2023-11-02T08:26:18.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 3/32 2023-11-02T08:26:18.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 4/32 2023-11-02T08:26:18.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/32 2023-11-02T08:26:18.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_6 6/32 2023-11-02T08:26:18.841 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 7/32 2023-11-02T08:26:18.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:18.2.0-1159.gb5a9486b.el9. 8/32 2023-11-02T08:26:18.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:18.2.0-1159.gb5a9486b.el9.x8 9/32 2023-11-02T08:26:18.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 10/32 2023-11-02T08:26:18.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 11/32 2023-11-02T08:26:18.842 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 12/32 2023-11-02T08:26:18.843 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 13/32 2023-11-02T08:26:18.843 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-11-02T08:26:18.843 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-11-02T08:26:18.843 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-11-02T08:26:18.843 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-11-02T08:26:18.844 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-02T08:26:18.844 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-11-02T08:26:18.844 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-11-02T08:26:18.844 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-11-02T08:26:18.844 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-11-02T08:26:18.845 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-11-02T08:26:18.845 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-11-02T08:26:18.845 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-11-02T08:26:18.845 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-11-02T08:26:18.845 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-11-02T08:26:18.846 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-11-02T08:26:18.846 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/32 2023-11-02T08:26:18.847 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-11-02T08:26:18.847 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 31/32 2023-11-02T08:26:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-11-02T08:26:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:19.838 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2023-11-02T08:26:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.838 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:26:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-02T08:26:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: fmt-8.1.1-5.el9.x86_64 2023-11-02T08:26:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-02T08:26:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-02T08:26:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-02T08:26:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-02T08:26:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-02T08:26:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-02T08:26:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-02T08:26:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-02T08:26:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-02T08:26:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-02T08:26:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-02T08:26:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: re2-1:20211101-3.el9.x86_64 2023-11-02T08:26:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-02T08:26:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:19.844 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:26:20.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test 2023-11-02T08:26:20.722 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:27 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:26:20.872 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:26:20.873 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:26:20.874 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:20.874 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:26:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 40 M 2023-11-02T08:26:20.874 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:26:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-11-02T08:26:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 166 k 2023-11-02T08:26:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-02T08:26:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-02T08:26:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-02T08:26:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:20.876 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:26:20.876 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:20.876 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2023-11-02T08:26:20.876 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:20.877 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 41 M 2023-11-02T08:26:20.877 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 167 M 2023-11-02T08:26:20.877 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:26:21.161 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-11-02T08:26:21.203 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-11-02T08:26:21.236 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-02T08:26:21.328 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): jq-1.6-15.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2023-11-02T08:26:21.362 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): libcephsqlite-18.2.0-1159.gb5a9486b.el9. 470 kB/s | 166 kB 00:00 2023-11-02T08:26:23.046 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): ceph-test-18.2.0-1159.gb5a9486b.el9.x86_ 19 MB/s | 40 MB 00:02 2023-11-02T08:26:23.049 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:23.049 INFO:teuthology.orchestra.run.smithi151.stdout:Total 19 MB/s | 41 MB 00:02 2023-11-02T08:26:23.063 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:26:23.084 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:26:23.085 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:26:23.417 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:26:23.417 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:26:23.967 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:26:24.067 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-11-02T08:26:24.108 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-11-02T08:26:24.138 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-02T08:26:24.181 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-11-02T08:26:24.208 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/6 2023-11-02T08:26:27.995 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/32 2023-11-02T08:26:27.996 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2023-11-02T08:26:27.996 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2023-11-02T08:26:27.996 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2023-11-02T08:26:27.997 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2023-11-02T08:26:27.997 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2023-11-02T08:26:27.997 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /root 2023-11-02T08:26:27.998 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2023-11-02T08:26:27.998 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:28.110 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/6 2023-11-02T08:26:28.225 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 6/6 2023-11-02T08:26:28.763 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/32 2023-11-02T08:26:28.788 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/32 2023-11-02T08:26:28.788 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:28.789 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-02T08:26:28.789 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-02T08:26:28.789 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-02T08:26:28.790 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:28.792 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-02T08:26:28.865 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-02T08:26:28.866 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-02T08:26:30.049 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-02T08:26:30.049 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/32 2023-11-02T08:26:30.049 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/32 2023-11-02T08:26:30.050 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 3/32 2023-11-02T08:26:30.050 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 4/32 2023-11-02T08:26:30.050 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/32 2023-11-02T08:26:30.050 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_6 6/32 2023-11-02T08:26:30.051 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 7/32 2023-11-02T08:26:30.051 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:18.2.0-1159.gb5a9486b.el9. 8/32 2023-11-02T08:26:30.051 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:18.2.0-1159.gb5a9486b.el9.x8 9/32 2023-11-02T08:26:30.052 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 10/32 2023-11-02T08:26:30.052 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 11/32 2023-11-02T08:26:30.052 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 12/32 2023-11-02T08:26:30.052 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 13/32 2023-11-02T08:26:30.052 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-11-02T08:26:30.053 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-11-02T08:26:30.053 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-11-02T08:26:30.053 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-11-02T08:26:30.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-02T08:26:30.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-11-02T08:26:30.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-11-02T08:26:30.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-11-02T08:26:30.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-11-02T08:26:30.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-11-02T08:26:30.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-11-02T08:26:30.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-11-02T08:26:30.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-11-02T08:26:30.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-11-02T08:26:30.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-11-02T08:26:30.057 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/32 2023-11-02T08:26:30.057 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-11-02T08:26:30.058 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 31/32 2023-11-02T08:26:30.403 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 6/6 2023-11-02T08:26:30.403 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/6 2023-11-02T08:26:30.403 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/6 2023-11-02T08:26:30.404 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-02T08:26:30.404 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-11-02T08:26:30.404 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-11-02T08:26:30.776 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-11-02T08:26:30.776 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:30.776 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:26:30.777 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:30.777 INFO:teuthology.orchestra.run.smithi151.stdout: jq-1.6-15.el9.x86_64 2023-11-02T08:26:30.777 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:30.777 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-02T08:26:30.777 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-02T08:26:30.778 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-02T08:26:30.778 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:30.778 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:26:31.058 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-11-02T08:26:31.058 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:31.058 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2023-11-02T08:26:31.059 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.059 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.059 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:26:31.059 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-02T08:26:31.059 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.060 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.060 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.060 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.060 INFO:teuthology.orchestra.run.smithi178.stdout: fmt-8.1.1-5.el9.x86_64 2023-11-02T08:26:31.060 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-02T08:26:31.060 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-02T08:26:31.061 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-02T08:26:31.061 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.061 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-02T08:26:31.061 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-02T08:26:31.061 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-02T08:26:31.062 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.062 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-02T08:26:31.062 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.062 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-02T08:26:31.062 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-02T08:26:31.063 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-02T08:26:31.063 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.063 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.063 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.063 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-02T08:26:31.064 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.064 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.064 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:31.064 INFO:teuthology.orchestra.run.smithi178.stdout: re2-1:20211101-3.el9.x86_64 2023-11-02T08:26:31.064 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-02T08:26:31.065 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:31.065 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:26:31.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph 2023-11-02T08:26:31.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test 2023-11-02T08:26:31.635 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:38 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:26:31.779 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:26:31.781 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2023-11-02T08:26:31.782 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:26:31.782 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2023-11-02T08:26:31.782 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:26:31.782 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 6.5 k 2023-11-02T08:26:31.783 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:26:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 2.1 M 2023-11-02T08:26:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 1.4 M 2023-11-02T08:26:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 242 k 2023-11-02T08:26:31.783 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 4.4 M 2023-11-02T08:26:31.784 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 18 M 2023-11-02T08:26:31.784 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-02T08:26:31.784 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-02T08:26:31.784 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-02T08:26:31.785 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-02T08:26:31.786 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-11-02T08:26:31.786 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-02T08:26:31.786 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-02T08:26:31.786 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-02T08:26:31.786 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-02T08:26:31.787 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-02T08:26:31.787 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-02T08:26:31.787 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-02T08:26:31.787 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-02T08:26:31.787 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-02T08:26:31.788 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-02T08:26:31.788 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-02T08:26:31.788 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-02T08:26:31.788 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-02T08:26:31.788 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-02T08:26:31.789 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-11-02T08:26:31.789 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-02T08:26:31.789 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-02T08:26:31.789 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-02T08:26:31.790 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-02T08:26:31.790 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-02T08:26:31.790 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-02T08:26:31.790 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-02T08:26:31.790 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:31.791 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:26:31.791 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2023-11-02T08:26:31.791 INFO:teuthology.orchestra.run.smithi151.stdout:Install 33 Packages 2023-11-02T08:26:31.791 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:31.792 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 30 M 2023-11-02T08:26:31.792 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 109 M 2023-11-02T08:26:31.792 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:26:31.908 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:28 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:26:32.053 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:26:32.054 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:32.054 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:26:32.054 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:32.055 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:26:32.055 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 40 M 2023-11-02T08:26:32.055 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:26:32.055 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-11-02T08:26:32.055 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 166 k 2023-11-02T08:26:32.055 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-02T08:26:32.056 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-02T08:26:32.056 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-02T08:26:32.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:32.056 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:26:32.056 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:32.057 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2023-11-02T08:26:32.057 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:32.057 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 41 M 2023-11-02T08:26:32.058 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 167 M 2023-11-02T08:26:32.058 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:26:32.245 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-11-02T08:26:32.290 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2023-11-02T08:26:32.320 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-02T08:26:32.454 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): jq-1.6-15.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-11-02T08:26:32.478 INFO:teuthology.orchestra.run.smithi151.stdout:(1/33): ceph-18.2.0-1159.gb5a9486b.el9.x86_64.r 38 kB/s | 6.5 kB 00:00 2023-11-02T08:26:32.487 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): libcephsqlite-18.2.0-1159.gb5a9486b.el9. 425 kB/s | 166 kB 00:00 2023-11-02T08:26:32.754 INFO:teuthology.orchestra.run.smithi151.stdout:(2/33): ceph-mgr-18.2.0-1159.gb5a9486b.el9.x86_ 3.3 MB/s | 1.4 MB 00:00 2023-11-02T08:26:32.862 INFO:teuthology.orchestra.run.smithi151.stdout:(3/33): ceph-mds-18.2.0-1159.gb5a9486b.el9.x86_ 3.8 MB/s | 2.1 MB 00:00 2023-11-02T08:26:32.929 INFO:teuthology.orchestra.run.smithi151.stdout:(4/33): ceph-mgr-modules-core-18.2.0-1159.gb5a9 3.6 MB/s | 242 kB 00:00 2023-11-02T08:26:33.030 INFO:teuthology.orchestra.run.smithi151.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-11-02T08:26:33.064 INFO:teuthology.orchestra.run.smithi151.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2023-11-02T08:26:33.098 INFO:teuthology.orchestra.run.smithi151.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-11-02T08:26:33.140 INFO:teuthology.orchestra.run.smithi151.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-11-02T08:26:33.198 INFO:teuthology.orchestra.run.smithi151.stdout:(9/33): ceph-mon-18.2.0-1159.gb5a9486b.el9.x86_ 6.1 MB/s | 4.4 MB 00:00 2023-11-02T08:26:33.232 INFO:teuthology.orchestra.run.smithi151.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2023-11-02T08:26:33.258 INFO:teuthology.orchestra.run.smithi151.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2023-11-02T08:26:33.283 INFO:teuthology.orchestra.run.smithi151.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-11-02T08:26:33.308 INFO:teuthology.orchestra.run.smithi151.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-11-02T08:26:33.367 INFO:teuthology.orchestra.run.smithi151.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 874 kB/s | 51 kB 00:00 2023-11-02T08:26:33.659 INFO:teuthology.orchestra.run.smithi151.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 149 kB/s | 43 kB 00:00 2023-11-02T08:26:33.851 INFO:teuthology.orchestra.run.smithi151.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 896 kB/s | 172 kB 00:00 2023-11-02T08:26:34.018 INFO:teuthology.orchestra.run.smithi151.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2023-11-02T08:26:34.085 INFO:teuthology.orchestra.run.smithi151.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2023-11-02T08:26:34.152 INFO:teuthology.orchestra.run.smithi151.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 266 kB/s | 18 kB 00:00 2023-11-02T08:26:34.219 INFO:teuthology.orchestra.run.smithi151.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2023-11-02T08:26:34.286 INFO:teuthology.orchestra.run.smithi151.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2023-11-02T08:26:34.338 INFO:teuthology.orchestra.run.smithi151.stdout:(22/33): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:01 2023-11-02T08:26:34.364 INFO:teuthology.orchestra.run.smithi151.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 254 kB/s | 20 kB 00:00 2023-11-02T08:26:34.456 INFO:teuthology.orchestra.run.smithi151.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 858 kB/s | 79 kB 00:00 2023-11-02T08:26:34.640 INFO:teuthology.orchestra.run.smithi151.stdout:(25/33): ceph-osd-18.2.0-1159.gb5a9486b.el9.x86 9.3 MB/s | 18 MB 00:01 2023-11-02T08:26:34.665 INFO:teuthology.orchestra.run.smithi151.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 142 kB/s | 46 kB 00:00 2023-11-02T08:26:34.690 INFO:teuthology.orchestra.run.smithi151.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2023-11-02T08:26:34.782 INFO:teuthology.orchestra.run.smithi151.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2023-11-02T08:26:34.833 INFO:teuthology.orchestra.run.smithi151.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 536 kB/s | 90 kB 00:00 2023-11-02T08:26:34.875 INFO:teuthology.orchestra.run.smithi151.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.5 MB/s | 230 kB 00:00 2023-11-02T08:26:34.909 INFO:teuthology.orchestra.run.smithi151.stdout:(31/33): python3-portend-3.1.0-2.el9.noarch.rpm 61 kB/s | 16 kB 00:00 2023-11-02T08:26:34.950 INFO:teuthology.orchestra.run.smithi151.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2023-11-02T08:26:35.034 INFO:teuthology.orchestra.run.smithi151.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.1 MB/s | 427 kB 00:00 2023-11-02T08:26:35.039 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:35.039 INFO:teuthology.orchestra.run.smithi151.stdout:Total 9.2 MB/s | 30 MB 00:03 2023-11-02T08:26:35.069 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:26:35.104 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:26:35.105 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:26:35.129 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): ceph-test-18.2.0-1159.gb5a9486b.el9.x86_ 13 MB/s | 40 MB 00:03 2023-11-02T08:26:35.133 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:35.134 INFO:teuthology.orchestra.run.smithi178.stdout:Total 13 MB/s | 41 MB 00:03 2023-11-02T08:26:35.147 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:26:35.168 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:26:35.168 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:26:35.460 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:26:35.460 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:26:35.509 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:26:35.510 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:26:36.036 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:26:36.121 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-11-02T08:26:36.157 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-11-02T08:26:36.187 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-02T08:26:36.230 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-11-02T08:26:36.257 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/6 2023-11-02T08:26:36.709 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:26:36.771 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-02T08:26:36.803 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-02T08:26:36.844 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-02T08:26:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-02T08:26:37.022 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-02T08:26:37.064 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-02T08:26:37.102 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-02T08:26:37.170 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-11-02T08:26:37.225 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-11-02T08:26:37.271 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-11-02T08:26:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-11-02T08:26:37.399 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-02T08:26:37.601 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-11-02T08:26:37.677 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-11-02T08:26:37.715 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-11-02T08:26:37.750 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-11-02T08:26:37.781 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-11-02T08:26:37.856 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-11-02T08:26:37.891 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-11-02T08:26:37.925 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-11-02T08:26:37.967 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-11-02T08:26:38.002 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-02T08:26:38.098 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-02T08:26:38.141 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-02T08:26:38.222 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-02T08:26:38.366 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-02T08:26:38.483 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-02T08:26:38.648 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-1159.gb5a9486b.el9. 28/33 2023-11-02T08:26:38.674 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/33 2023-11-02T08:26:38.697 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/33 2023-11-02T08:26:38.698 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:38.698 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-02T08:26:38.698 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-02T08:26:38.698 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-02T08:26:38.699 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:39.986 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/6 2023-11-02T08:26:40.063 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 6/6 2023-11-02T08:26:40.279 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/33 2023-11-02T08:26:40.307 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/33 2023-11-02T08:26:40.307 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:40.307 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-02T08:26:40.308 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-02T08:26:40.308 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-02T08:26:40.308 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:40.755 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 31/33 2023-11-02T08:26:40.778 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 31/33 2023-11-02T08:26:40.778 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:40.779 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-02T08:26:40.779 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-02T08:26:40.779 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-02T08:26:40.779 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:40.989 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 32/33 2023-11-02T08:26:41.012 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 32/33 2023-11-02T08:26:41.012 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:41.012 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-02T08:26:41.013 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-02T08:26:41.013 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-02T08:26:41.013 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:41.078 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 33/33 2023-11-02T08:26:42.234 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 6/6 2023-11-02T08:26:42.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/6 2023-11-02T08:26:42.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/6 2023-11-02T08:26:42.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-02T08:26:42.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-11-02T08:26:42.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-11-02T08:26:42.599 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-11-02T08:26:42.600 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:42.600 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:26:42.600 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:42.600 INFO:teuthology.orchestra.run.smithi178.stdout: jq-1.6-15.el9.x86_64 2023-11-02T08:26:42.601 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:42.601 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-02T08:26:42.601 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-02T08:26:42.601 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-02T08:26:42.601 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:42.602 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:26:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 33/33 2023-11-02T08:26:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/33 2023-11-02T08:26:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/33 2023-11-02T08:26:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 3/33 2023-11-02T08:26:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 4/33 2023-11-02T08:26:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/33 2023-11-02T08:26:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-1159.gb5a9486b.el9. 6/33 2023-11-02T08:26:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-02T08:26:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-02T08:26:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-11-02T08:26:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-11-02T08:26:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-11-02T08:26:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-02T08:26:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-11-02T08:26:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-11-02T08:26:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-11-02T08:26:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-11-02T08:26:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-02T08:26:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-02T08:26:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-02T08:26:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-02T08:26:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-02T08:26:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-02T08:26:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-02T08:26:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-02T08:26:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-02T08:26:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-02T08:26:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-02T08:26:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-02T08:26:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-02T08:26:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-02T08:26:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-02T08:26:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-02T08:26:42.873 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph 2023-11-02T08:26:43.420 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:40 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:26:43.565 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:26:43.567 INFO:teuthology.orchestra.run.smithi178.stdout:========================================================================================= 2023-11-02T08:26:43.568 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:26:43.568 INFO:teuthology.orchestra.run.smithi178.stdout:========================================================================================= 2023-11-02T08:26:43.568 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:26:43.568 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 6.5 k 2023-11-02T08:26:43.569 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:26:43.569 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 2.1 M 2023-11-02T08:26:43.569 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 1.4 M 2023-11-02T08:26:43.569 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 242 k 2023-11-02T08:26:43.569 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 4.4 M 2023-11-02T08:26:43.570 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 18 M 2023-11-02T08:26:43.570 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-02T08:26:43.570 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-02T08:26:43.570 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-02T08:26:43.570 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-02T08:26:43.571 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-11-02T08:26:43.571 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-02T08:26:43.571 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-02T08:26:43.571 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-02T08:26:43.571 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-02T08:26:43.572 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-02T08:26:43.572 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-02T08:26:43.572 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-02T08:26:43.572 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-02T08:26:43.572 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-02T08:26:43.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-02T08:26:43.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-02T08:26:43.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-02T08:26:43.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-02T08:26:43.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-02T08:26:43.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-11-02T08:26:43.574 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-02T08:26:43.574 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-02T08:26:43.574 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-02T08:26:43.574 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-02T08:26:43.574 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-02T08:26:43.575 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-02T08:26:43.575 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-02T08:26:43.575 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:43.575 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:26:43.575 INFO:teuthology.orchestra.run.smithi178.stdout:========================================================================================= 2023-11-02T08:26:43.576 INFO:teuthology.orchestra.run.smithi178.stdout:Install 33 Packages 2023-11-02T08:26:43.576 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:43.577 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 30 M 2023-11-02T08:26:43.577 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 109 M 2023-11-02T08:26:43.577 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:26:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-02T08:26:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:43.773 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:26:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:26:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-02T08:26:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-02T08:26:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-02T08:26:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-02T08:26:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-02T08:26:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-02T08:26:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-02T08:26:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-02T08:26:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-02T08:26:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-02T08:26:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-02T08:26:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-02T08:26:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-02T08:26:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-02T08:26:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-02T08:26:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2023-11-02T08:26:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-02T08:26:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-02T08:26:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-02T08:26:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-02T08:26:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-02T08:26:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-02T08:26:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-02T08:26:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-02T08:26:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-02T08:26:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-02T08:26:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-02T08:26:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:43.781 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:26:43.939 INFO:teuthology.orchestra.run.smithi178.stdout:(1/33): ceph-18.2.0-1159.gb5a9486b.el9.x86_64.r 32 kB/s | 6.5 kB 00:00 2023-11-02T08:26:44.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base 2023-11-02T08:26:44.256 INFO:teuthology.orchestra.run.smithi178.stdout:(2/33): ceph-mgr-18.2.0-1159.gb5a9486b.el9.x86_ 2.8 MB/s | 1.4 MB 00:00 2023-11-02T08:26:44.389 INFO:teuthology.orchestra.run.smithi178.stdout:(3/33): ceph-mds-18.2.0-1159.gb5a9486b.el9.x86_ 3.2 MB/s | 2.1 MB 00:00 2023-11-02T08:26:44.456 INFO:teuthology.orchestra.run.smithi178.stdout:(4/33): ceph-mgr-modules-core-18.2.0-1159.gb5a9 3.5 MB/s | 242 kB 00:00 2023-11-02T08:26:44.556 INFO:teuthology.orchestra.run.smithi178.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-11-02T08:26:44.582 INFO:teuthology.orchestra.run.smithi178.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-11-02T08:26:44.607 INFO:teuthology.orchestra.run.smithi178.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-11-02T08:26:44.649 INFO:teuthology.orchestra.run.smithi178.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-11-02T08:26:44.660 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:51 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:26:44.682 INFO:teuthology.orchestra.run.smithi178.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-11-02T08:26:44.716 INFO:teuthology.orchestra.run.smithi178.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-11-02T08:26:44.750 INFO:teuthology.orchestra.run.smithi178.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-11-02T08:26:44.751 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:26:44.792 INFO:teuthology.orchestra.run.smithi178.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2023-11-02T08:26:44.799 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:26:44.800 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:26:44.800 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:26:44.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install cephadm 2023-11-02T08:26:44.917 INFO:teuthology.orchestra.run.smithi178.stdout:(13/33): ceph-mon-18.2.0-1159.gb5a9486b.el9.x86 4.5 MB/s | 4.4 MB 00:00 2023-11-02T08:26:45.092 INFO:teuthology.orchestra.run.smithi178.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 337 kB/s | 51 kB 00:00 2023-11-02T08:26:45.142 INFO:teuthology.orchestra.run.smithi178.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 869 kB/s | 43 kB 00:00 2023-11-02T08:26:45.193 INFO:teuthology.orchestra.run.smithi178.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2023-11-02T08:26:45.235 INFO:teuthology.orchestra.run.smithi178.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 8.4 MB/s | 358 kB 00:00 2023-11-02T08:26:45.260 INFO:teuthology.orchestra.run.smithi178.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2023-11-02T08:26:45.285 INFO:teuthology.orchestra.run.smithi178.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 705 kB/s | 18 kB 00:00 2023-11-02T08:26:45.310 INFO:teuthology.orchestra.run.smithi178.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 927 kB/s | 23 kB 00:00 2023-11-02T08:26:45.336 INFO:teuthology.orchestra.run.smithi178.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 774 kB/s | 19 kB 00:00 2023-11-02T08:26:45.361 INFO:teuthology.orchestra.run.smithi178.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 784 kB/s | 20 kB 00:00 2023-11-02T08:26:45.386 INFO:teuthology.orchestra.run.smithi178.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2023-11-02T08:26:45.406 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:52 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:26:45.420 INFO:teuthology.orchestra.run.smithi178.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 2.3 MB/s | 79 kB 00:00 2023-11-02T08:26:45.454 INFO:teuthology.orchestra.run.smithi178.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2023-11-02T08:26:45.545 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:26:45.546 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:26:45.547 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:45.547 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:26:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 221 k 2023-11-02T08:26:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:45.547 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:26:45.548 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:45.548 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:26:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:45.548 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 221 k 2023-11-02T08:26:45.548 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 213 k 2023-11-02T08:26:45.548 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:26:45.637 INFO:teuthology.orchestra.run.smithi178.stdout:(26/33): ceph-osd-18.2.0-1159.gb5a9486b.el9.x86 13 MB/s | 18 MB 00:01 2023-11-02T08:26:45.663 INFO:teuthology.orchestra.run.smithi178.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 79 kB/s | 16 kB 00:00 2023-11-02T08:26:45.696 INFO:teuthology.orchestra.run.smithi178.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2023-11-02T08:26:45.730 INFO:teuthology.orchestra.run.smithi178.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 983 kB/s | 90 kB 00:00 2023-11-02T08:26:45.755 INFO:teuthology.orchestra.run.smithi178.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2023-11-02T08:26:45.789 INFO:teuthology.orchestra.run.smithi178.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 609 kB/s | 20 kB 00:00 2023-11-02T08:26:45.814 INFO:teuthology.orchestra.run.smithi178.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2023-11-02T08:26:45.818 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm-18.2.0-1159.gb5a9486b.el9.noarch.rpm 816 kB/s | 221 kB 00:00 2023-11-02T08:26:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:Total 811 kB/s | 221 kB 00:00 2023-11-02T08:26:45.820 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:26:45.824 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:26:45.825 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:26:45.827 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:26:45.827 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:26:45.864 INFO:teuthology.orchestra.run.smithi178.stdout:(33/33): python3-cryptography-36.0.1-4.el9.x86_ 1.1 MB/s | 1.2 MB 00:01 2023-11-02T08:26:45.868 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:45.868 INFO:teuthology.orchestra.run.smithi178.stdout:Total 13 MB/s | 30 MB 00:02 2023-11-02T08:26:45.897 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:26:45.899 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:26:45.936 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:26:45.936 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:26:46.311 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:26:46.312 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:26:46.347 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:26:46.440 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:26:46.800 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:26:47.091 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:26:47.091 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:47.092 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:26:47.092 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:26:47.092 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:47.092 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:26:47.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache 2023-11-02T08:26:47.530 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:26:47.592 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-02T08:26:47.632 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-02T08:26:47.666 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-02T08:26:47.755 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-02T08:26:47.835 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-02T08:26:47.884 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-02T08:26:47.888 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:54 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:26:47.922 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-02T08:26:47.989 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-11-02T08:26:48.033 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:26:48.034 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:48.035 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:26:48.035 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:48.035 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:26:48.036 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 142 k 2023-11-02T08:26:48.036 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:48.036 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:26:48.036 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:48.036 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:26:48.037 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:48.037 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 142 k 2023-11-02T08:26:48.037 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 400 k 2023-11-02T08:26:48.037 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:26:48.052 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-11-02T08:26:48.099 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-11-02T08:26:48.147 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-11-02T08:26:48.209 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-02T08:26:48.245 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-18.2.0-1159.gb5a948 681 kB/s | 142 kB 00:00 2023-11-02T08:26:48.246 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:48.246 INFO:teuthology.orchestra.run.smithi151.stdout:Total 676 kB/s | 142 kB 00:00 2023-11-02T08:26:48.246 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:26:48.253 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:26:48.253 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:26:48.305 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:26:48.305 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:26:48.418 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-11-02T08:26:48.454 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:26:48.470 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-11-02T08:26:48.500 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-11-02T08:26:48.513 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b. 1/1 2023-11-02T08:26:48.538 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b. 1/1 2023-11-02T08:26:48.539 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:48.539 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-02T08:26:48.539 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:48.551 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-11-02T08:26:48.581 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-11-02T08:26:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-11-02T08:26:48.692 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-11-02T08:26:48.725 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-11-02T08:26:48.766 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-11-02T08:26:48.793 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-02T08:26:48.880 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-02T08:26:48.915 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-02T08:26:48.995 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-02T08:26:49.140 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-02T08:26:49.264 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-02T08:26:49.436 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-1159.gb5a9486b.el9. 28/33 2023-11-02T08:26:49.462 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/33 2023-11-02T08:26:49.486 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/33 2023-11-02T08:26:49.487 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:49.487 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-02T08:26:49.487 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-02T08:26:49.488 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-02T08:26:49.488 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:49.507 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b. 1/1 2023-11-02T08:26:49.507 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:49.507 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:26:49.508 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:49.508 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:49.508 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:26:49.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr 2023-11-02T08:26:50.335 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:57 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:26:50.427 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:26:50.476 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:26:50.477 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:26:50.477 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:26:50.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-dashboard 2023-11-02T08:26:51.005 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/33 2023-11-02T08:26:51.034 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/33 2023-11-02T08:26:51.034 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:51.034 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-02T08:26:51.035 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-02T08:26:51.035 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-02T08:26:51.035 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:51.101 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:58 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 1.8 M 2023-11-02T08:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 23 k 2023-11-02T08:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 15 k 2023-11-02T08:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-02T08:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-02T08:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-02T08:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2023-11-02T08:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-02T08:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout:Install 7 Packages 2023-11-02T08:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.1 M 2023-11-02T08:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 8.5 M 2023-11-02T08:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:26:51.495 INFO:teuthology.orchestra.run.smithi151.stdout:(1/7): ceph-prometheus-alerts-18.2.0-1159.gb5a9 75 kB/s | 15 kB 00:00 2023-11-02T08:26:51.511 INFO:teuthology.orchestra.run.smithi151.stdout:(2/7): ceph-grafana-dashboards-18.2.0-1159.gb5a 109 kB/s | 23 kB 00:00 2023-11-02T08:26:51.514 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 31/33 2023-11-02T08:26:51.537 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 31/33 2023-11-02T08:26:51.537 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:51.538 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-02T08:26:51.538 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-02T08:26:51.538 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-02T08:26:51.538 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:51.603 INFO:teuthology.orchestra.run.smithi151.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 83 kB/s | 9.0 kB 00:00 2023-11-02T08:26:51.654 INFO:teuthology.orchestra.run.smithi151.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 288 kB/s | 41 kB 00:00 2023-11-02T08:26:51.712 INFO:teuthology.orchestra.run.smithi151.stdout:(5/7): ceph-mgr-dashboard-18.2.0-1159.gb5a9486b 4.2 MB/s | 1.8 MB 00:00 2023-11-02T08:26:51.738 INFO:teuthology.orchestra.run.smithi151.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 229 kB/s | 31 kB 00:00 2023-11-02T08:26:51.739 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 32/33 2023-11-02T08:26:51.763 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 32/33 2023-11-02T08:26:51.763 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:51.764 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-02T08:26:51.764 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-02T08:26:51.764 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-02T08:26:51.764 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:51.838 INFO:teuthology.orchestra.run.smithi151.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2023-11-02T08:26:51.839 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:51.840 INFO:teuthology.orchestra.run.smithi151.stdout:Total 3.5 MB/s | 2.1 MB 00:00 2023-11-02T08:26:51.840 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:26:51.848 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:26:51.848 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:26:51.849 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 33/33 2023-11-02T08:26:51.914 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:26:51.914 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:26:52.211 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:26:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-02T08:26:52.315 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-02T08:26:52.370 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-02T08:26:52.402 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-02T08:26:52.439 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-1159.gb5a9486b.el9.n 5/7 2023-11-02T08:26:52.699 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-1159.gb5a9486b.el9. 6/7 2023-11-02T08:26:52.743 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarc 7/7 2023-11-02T08:26:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarc 7/7 2023-11-02T08:26:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-1159.gb5a9486b.el9. 1/7 2023-11-02T08:26:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarc 2/7 2023-11-02T08:26:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-1159.gb5a9486b.el9.n 3/7 2023-11-02T08:26:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-02T08:26:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-02T08:26:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-02T08:26:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-02T08:26:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:53.622 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:26:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:26:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:26:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:26:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-02T08:26:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-02T08:26:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-02T08:26:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-02T08:26:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:53.624 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:26:53.681 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 33/33 2023-11-02T08:26:53.681 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/33 2023-11-02T08:26:53.682 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/33 2023-11-02T08:26:53.682 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 3/33 2023-11-02T08:26:53.682 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 4/33 2023-11-02T08:26:53.682 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/33 2023-11-02T08:26:53.683 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-1159.gb5a9486b.el9. 6/33 2023-11-02T08:26:53.683 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-02T08:26:53.683 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-02T08:26:53.683 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-11-02T08:26:53.684 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-11-02T08:26:53.684 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-11-02T08:26:53.684 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-02T08:26:53.684 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-11-02T08:26:53.684 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-11-02T08:26:53.685 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-11-02T08:26:53.685 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-11-02T08:26:53.685 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-02T08:26:53.686 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-02T08:26:53.686 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-02T08:26:53.686 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-02T08:26:53.687 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-02T08:26:53.687 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-02T08:26:53.687 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-02T08:26:53.688 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-02T08:26:53.688 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-02T08:26:53.688 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-02T08:26:53.688 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-02T08:26:53.690 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-02T08:26:53.690 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-02T08:26:53.690 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-02T08:26:53.691 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-02T08:26:53.691 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-02T08:26:53.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-02T08:26:54.426 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:01 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:26:54.496 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-02T08:26:54.496 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:54.496 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:26:54.496 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:54.497 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:54.497 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:54.497 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:26:54.497 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:54.497 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:26:54.498 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-02T08:26:54.498 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-02T08:26:54.498 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-02T08:26:54.498 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-02T08:26:54.498 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-02T08:26:54.499 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-02T08:26:54.499 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-02T08:26:54.499 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-02T08:26:54.499 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-02T08:26:54.499 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-02T08:26:54.500 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-02T08:26:54.500 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-02T08:26:54.500 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-02T08:26:54.500 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-02T08:26:54.500 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-02T08:26:54.500 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply-3.11-14.el9.noarch 2023-11-02T08:26:54.501 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-02T08:26:54.501 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-02T08:26:54.501 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-02T08:26:54.501 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-02T08:26:54.501 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-02T08:26:54.502 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-02T08:26:54.502 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-02T08:26:54.502 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-02T08:26:54.502 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-02T08:26:54.502 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-02T08:26:54.502 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-02T08:26:54.503 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:54.503 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:26:54.569 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:26:54.570 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2023-11-02T08:26:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:26:54.571 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2023-11-02T08:26:54.571 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:26:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 7.4 M 2023-11-02T08:26:54.571 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:26:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-02T08:26:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-02T08:26:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-02T08:26:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-02T08:26:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-02T08:26:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-02T08:26:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-02T08:26:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-11-02T08:26:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-02T08:26:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-02T08:26:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-02T08:26:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:54.574 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:26:54.574 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2023-11-02T08:26:54.574 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2023-11-02T08:26:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:26:54.575 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 37 M 2023-11-02T08:26:54.575 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 209 M 2023-11-02T08:26:54.575 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:26:54.722 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 212 kB/s | 32 kB 00:00 2023-11-02T08:26:54.756 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 529 kB/s | 17 kB 00:00 2023-11-02T08:26:54.789 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-11-02T08:26:54.834 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base 2023-11-02T08:26:54.839 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-11-02T08:26:54.932 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 34 MB/s | 4.7 MB 00:00 2023-11-02T08:26:54.957 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 205 kB 00:00 2023-11-02T08:26:55.032 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2023-11-02T08:26:55.058 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-11-02T08:26:55.116 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2023-11-02T08:26:55.300 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-11-02T08:26:55.325 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 945 kB/s | 195 kB 00:00 2023-11-02T08:26:55.379 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:52 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:26:55.470 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:26:55.518 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:26:55.519 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:26:55.519 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:26:55.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install cephadm 2023-11-02T08:26:55.634 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.0-1 7.0 MB/s | 7.4 MB 00:01 2023-11-02T08:26:55.637 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:55.637 INFO:teuthology.orchestra.run.smithi151.stdout:Total 35 MB/s | 37 MB 00:01 2023-11-02T08:26:55.638 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:26:55.660 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:26:55.660 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:26:56.091 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:26:56.091 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:26:56.120 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:53 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:26:56.260 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:26:56.261 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:56.261 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:26:56.261 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:56.261 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:26:56.261 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 221 k 2023-11-02T08:26:56.262 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:56.262 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:26:56.262 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:56.262 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:26:56.262 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:56.263 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 221 k 2023-11-02T08:26:56.263 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 213 k 2023-11-02T08:26:56.263 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:26:56.521 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm-18.2.0-1159.gb5a9486b.el9.noarch.rpm 854 kB/s | 221 kB 00:00 2023-11-02T08:26:56.522 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:56.522 INFO:teuthology.orchestra.run.smithi178.stdout:Total 849 kB/s | 221 kB 00:00 2023-11-02T08:26:56.522 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:26:56.527 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:26:56.527 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:26:56.529 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:26:56.530 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:26:56.598 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:26:56.817 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:26:56.891 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-02T08:26:56.922 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-02T08:26:57.015 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-02T08:26:57.048 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:26:57.051 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-02T08:26:57.133 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:26:57.228 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-02T08:26:57.258 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-02T08:26:57.332 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-02T08:26:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:26:57.814 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:26:57.814 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:57.815 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:26:57.815 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:26:57.815 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:57.816 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:26:58.005 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-02T08:26:58.032 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache 2023-11-02T08:26:58.114 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-02T08:26:58.583 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:55 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:26:58.730 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:26:58.731 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:58.731 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:26:58.731 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:58.732 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:26:58.732 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 142 k 2023-11-02T08:26:58.732 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:58.732 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:26:58.733 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:26:58.733 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:26:58.733 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:58.733 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 142 k 2023-11-02T08:26:58.733 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 400 k 2023-11-02T08:26:58.734 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:26:58.977 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-18.2.0-1159.gb5a948 582 kB/s | 142 kB 00:00 2023-11-02T08:26:58.978 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:26:58.978 INFO:teuthology.orchestra.run.smithi178.stdout:Total 579 kB/s | 142 kB 00:00 2023-11-02T08:26:58.978 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:26:58.985 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:26:58.985 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:26:59.037 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:26:59.038 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:26:59.173 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:26:59.244 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b. 1/1 2023-11-02T08:26:59.269 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b. 1/1 2023-11-02T08:26:59.269 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:26:59.269 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-02T08:26:59.270 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:26:59.347 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-02T08:27:00.205 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b. 1/1 2023-11-02T08:27:00.205 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:00.205 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:00.205 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:00.206 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:00.206 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:00.369 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-02T08:27:00.388 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a94 12/12 2023-11-02T08:27:00.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr 2023-11-02T08:27:01.004 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:58 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:01.095 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:01.145 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:01.145 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:27:01.145 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:01.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-dashboard 2023-11-02T08:27:01.757 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:58 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:01.902 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:01.903 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:01.903 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:27:01.903 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:01.903 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:01.904 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 1.8 M 2023-11-02T08:27:01.904 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:27:01.904 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 23 k 2023-11-02T08:27:01.904 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 15 k 2023-11-02T08:27:01.904 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-02T08:27:01.905 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-02T08:27:01.905 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-02T08:27:01.905 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2023-11-02T08:27:01.905 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-02T08:27:01.905 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:01.906 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:01.906 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:01.906 INFO:teuthology.orchestra.run.smithi178.stdout:Install 7 Packages 2023-11-02T08:27:01.906 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:01.907 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 2.1 M 2023-11-02T08:27:01.907 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 8.5 M 2023-11-02T08:27:01.907 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:02.322 INFO:teuthology.orchestra.run.smithi178.stdout:(1/7): ceph-grafana-dashboards-18.2.0-1159.gb5a 69 kB/s | 23 kB 00:00 2023-11-02T08:27:02.347 INFO:teuthology.orchestra.run.smithi178.stdout:(2/7): ceph-prometheus-alerts-18.2.0-1159.gb5a9 40 kB/s | 15 kB 00:00 2023-11-02T08:27:02.389 INFO:teuthology.orchestra.run.smithi178.stdout:(3/7): ceph-mgr-dashboard-18.2.0-1159.gb5a9486b 4.3 MB/s | 1.8 MB 00:00 2023-11-02T08:27:02.539 INFO:teuthology.orchestra.run.smithi178.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 42 kB/s | 9.0 kB 00:00 2023-11-02T08:27:02.657 INFO:teuthology.orchestra.run.smithi178.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 115 kB/s | 31 kB 00:00 2023-11-02T08:27:02.690 INFO:teuthology.orchestra.run.smithi178.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 119 kB/s | 41 kB 00:00 2023-11-02T08:27:02.882 INFO:teuthology.orchestra.run.smithi178.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 551 kB/s | 188 kB 00:00 2023-11-02T08:27:02.883 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:02.884 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.1 MB/s | 2.1 MB 00:00 2023-11-02T08:27:02.884 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:02.892 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:02.892 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:02.962 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:02.962 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a94 12/12 2023-11-02T08:27:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a94 1/12 2023-11-02T08:27:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-02T08:27:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-02T08:27:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-02T08:27:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-02T08:27:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-02T08:27:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-11-02T08:27:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-11-02T08:27:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-11-02T08:27:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-11-02T08:27:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-11-02T08:27:03.250 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:03.308 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-02T08:27:03.341 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-02T08:27:03.388 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-02T08:27:03.430 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-02T08:27:03.458 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-1159.gb5a9486b.el9.n 5/7 2023-11-02T08:27:03.690 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-1159.gb5a9486b.el9. 6/7 2023-11-02T08:27:03.701 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-11-02T08:27:03.701 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:03.702 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:03.702 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:03.702 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-02T08:27:03.702 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-02T08:27:03.703 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-02T08:27:03.703 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-02T08:27:03.703 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-02T08:27:03.703 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-02T08:27:03.703 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-02T08:27:03.704 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-02T08:27:03.704 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-02T08:27:03.704 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-02T08:27:03.704 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-02T08:27:03.704 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:03.705 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:03.720 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarc 7/7 2023-11-02T08:27:04.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-rook 2023-11-02T08:27:04.231 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarc 7/7 2023-11-02T08:27:04.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-1159.gb5a9486b.el9. 1/7 2023-11-02T08:27:04.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarc 2/7 2023-11-02T08:27:04.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-1159.gb5a9486b.el9.n 3/7 2023-11-02T08:27:04.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-02T08:27:04.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-02T08:27:04.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-02T08:27:04.547 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-02T08:27:04.548 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:04.548 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:04.548 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:04.549 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:04.549 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:04.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-02T08:27:04.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-02T08:27:04.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-02T08:27:04.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-02T08:27:04.550 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:04.550 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:04.648 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:11 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:04.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-02T08:27:04.791 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:04.792 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2023-11-02T08:27:04.792 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:27:04.793 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2023-11-02T08:27:04.793 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:04.793 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 47 k 2023-11-02T08:27:04.793 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:27:04.793 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-02T08:27:04.794 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-02T08:27:04.794 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-11-02T08:27:04.794 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-02T08:27:04.794 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-02T08:27:04.795 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-02T08:27:04.795 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-02T08:27:04.795 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-02T08:27:04.795 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-02T08:27:04.795 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-02T08:27:04.796 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-02T08:27:04.796 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-02T08:27:04.796 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:04.796 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:04.796 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2023-11-02T08:27:04.797 INFO:teuthology.orchestra.run.smithi151.stdout:Install 13 Packages 2023-11-02T08:27:04.797 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:04.797 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.0 M 2023-11-02T08:27:04.798 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2023-11-02T08:27:04.798 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:04.977 INFO:teuthology.orchestra.run.smithi151.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 210 kB/s | 24 kB 00:00 2023-11-02T08:27:04.994 INFO:teuthology.orchestra.run.smithi151.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 18 kB 00:00 2023-11-02T08:27:05.044 INFO:teuthology.orchestra.run.smithi151.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-11-02T08:27:05.069 INFO:teuthology.orchestra.run.smithi151.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-11-02T08:27:05.095 INFO:teuthology.orchestra.run.smithi151.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-11-02T08:27:05.120 INFO:teuthology.orchestra.run.smithi151.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 851 kB/s | 43 kB 00:00 2023-11-02T08:27:05.146 INFO:teuthology.orchestra.run.smithi151.stdout:(7/13): ceph-mgr-rook-18.2.0-1159.gb5a9486b.el9 166 kB/s | 47 kB 00:00 2023-11-02T08:27:05.188 INFO:teuthology.orchestra.run.smithi151.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 224 kB/s | 15 kB 00:00 2023-11-02T08:27:05.213 INFO:teuthology.orchestra.run.smithi151.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 273 kB/s | 32 kB 00:00 2023-11-02T08:27:05.255 INFO:teuthology.orchestra.run.smithi151.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2023-11-02T08:27:05.281 INFO:teuthology.orchestra.run.smithi151.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 1.4 MB/s | 197 kB 00:00 2023-11-02T08:27:05.306 INFO:teuthology.orchestra.run.smithi151.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2023-11-02T08:27:05.314 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:02 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:05.356 INFO:teuthology.orchestra.run.smithi151.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2023-11-02T08:27:05.357 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:05.358 INFO:teuthology.orchestra.run.smithi151.stdout:Total 3.6 MB/s | 2.0 MB 00:00 2023-11-02T08:27:05.358 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:05.371 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:05.371 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:05.458 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:05.459 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2023-11-02T08:27:05.459 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:27:05.460 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2023-11-02T08:27:05.460 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:05.460 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 7.4 M 2023-11-02T08:27:05.460 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:27:05.460 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-02T08:27:05.461 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-02T08:27:05.461 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-02T08:27:05.461 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-02T08:27:05.461 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-02T08:27:05.461 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-02T08:27:05.462 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-02T08:27:05.462 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-11-02T08:27:05.462 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-02T08:27:05.462 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-02T08:27:05.462 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-02T08:27:05.462 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:05.463 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:05.463 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2023-11-02T08:27:05.463 INFO:teuthology.orchestra.run.smithi178.stdout:Install 12 Packages 2023-11-02T08:27:05.463 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:05.464 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 37 M 2023-11-02T08:27:05.464 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 209 M 2023-11-02T08:27:05.465 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:05.466 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:05.468 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:05.613 INFO:teuthology.orchestra.run.smithi178.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 209 kB/s | 32 kB 00:00 2023-11-02T08:27:05.647 INFO:teuthology.orchestra.run.smithi178.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 527 kB/s | 17 kB 00:00 2023-11-02T08:27:05.680 INFO:teuthology.orchestra.run.smithi178.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-11-02T08:27:05.739 INFO:teuthology.orchestra.run.smithi178.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-11-02T08:27:05.831 INFO:teuthology.orchestra.run.smithi178.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 205 kB 00:00 2023-11-02T08:27:05.898 INFO:teuthology.orchestra.run.smithi178.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2023-11-02T08:27:05.973 INFO:teuthology.orchestra.run.smithi178.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2023-11-02T08:27:05.993 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:05.998 INFO:teuthology.orchestra.run.smithi178.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-11-02T08:27:06.153 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-02T08:27:06.216 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-02T08:27:06.274 INFO:teuthology.orchestra.run.smithi178.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 52 MB/s | 16 MB 00:00 2023-11-02T08:27:06.280 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-02T08:27:06.310 INFO:teuthology.orchestra.run.smithi178.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2023-11-02T08:27:06.318 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-02T08:27:06.332 INFO:teuthology.orchestra.run.smithi178.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2023-11-02T08:27:06.359 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-02T08:27:06.453 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-02T08:27:06.563 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-11-02T08:27:06.620 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-02T08:27:06.666 INFO:teuthology.orchestra.run.smithi178.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.0-1 6.1 MB/s | 7.4 MB 00:01 2023-11-02T08:27:06.668 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:06.669 INFO:teuthology.orchestra.run.smithi178.stdout:Total 31 MB/s | 37 MB 00:01 2023-11-02T08:27:06.669 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:06.691 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:06.691 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:07.120 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:07.121 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:07.341 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-02T08:27:07.393 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-02T08:27:07.427 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-02T08:27:07.473 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-02T08:27:07.497 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 13/13 2023-11-02T08:27:07.817 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:07.909 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-02T08:27:07.937 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-02T08:27:08.029 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-02T08:27:08.065 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-02T08:27:08.241 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-02T08:27:08.272 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-02T08:27:08.332 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 13/13 2023-11-02T08:27:08.333 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 1/13 2023-11-02T08:27:08.333 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-02T08:27:08.333 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-02T08:27:08.333 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-02T08:27:08.334 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-02T08:27:08.334 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-02T08:27:08.334 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-02T08:27:08.334 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-02T08:27:08.334 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-02T08:27:08.334 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-11-02T08:27:08.335 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-02T08:27:08.335 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-02T08:27:08.350 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-02T08:27:08.695 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-02T08:27:08.696 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:08.696 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:08.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:08.696 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-02T08:27:08.696 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-02T08:27:08.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-11-02T08:27:08.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-02T08:27:08.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-02T08:27:08.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-02T08:27:08.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-02T08:27:08.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-02T08:27:08.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-02T08:27:08.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-02T08:27:08.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-02T08:27:08.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-02T08:27:08.699 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:08.699 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:08.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-cephadm 2023-11-02T08:27:09.134 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-02T08:27:09.134 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-02T08:27:09.714 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:16 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:09.864 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:09.864 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2023-11-02T08:27:09.865 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:27:09.865 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2023-11-02T08:27:09.865 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:09.865 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 132 k 2023-11-02T08:27:09.865 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:27:09.866 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-02T08:27:09.866 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-02T08:27:09.866 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-02T08:27:09.866 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-02T08:27:09.867 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-02T08:27:09.867 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:09.867 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:09.867 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2023-11-02T08:27:09.868 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2023-11-02T08:27:09.869 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:09.869 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 6.8 M 2023-11-02T08:27:09.870 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 33 M 2023-11-02T08:27:09.870 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:10.280 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-11-02T08:27:10.335 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-02T08:27:10.397 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2023-11-02T08:27:10.581 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): ceph-mgr-cephadm-18.2.0-1159.gb5a9486b.e 294 kB/s | 132 kB 00:00 2023-11-02T08:27:10.689 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 215 kB/s | 58 kB 00:00 2023-11-02T08:27:10.756 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 291 kB/s | 51 kB 00:00 2023-11-02T08:27:10.865 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 939 kB/s | 548 kB 00:00 2023-11-02T08:27:10.868 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:10.868 INFO:teuthology.orchestra.run.smithi151.stdout:Total 6.8 MB/s | 6.8 MB 00:01 2023-11-02T08:27:10.869 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:10.878 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:10.878 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:10.967 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:10.968 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:11.283 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:11.368 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-02T08:27:11.405 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a94 12/12 2023-11-02T08:27:11.462 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-02T08:27:11.509 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-02T08:27:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-02T08:27:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-02T08:27:12.181 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-02T08:27:12.214 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 6/6 2023-11-02T08:27:13.160 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 6/6 2023-11-02T08:27:13.160 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/6 2023-11-02T08:27:13.160 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-02T08:27:13.160 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-02T08:27:13.161 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-02T08:27:13.161 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-02T08:27:13.478 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-02T08:27:13.478 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:13.478 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:13.479 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:13.479 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-02T08:27:13.479 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-02T08:27:13.479 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-02T08:27:13.479 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-02T08:27:13.480 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-02T08:27:13.480 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:13.480 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:13.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse 2023-11-02T08:27:13.880 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a94 12/12 2023-11-02T08:27:13.880 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a94 1/12 2023-11-02T08:27:13.881 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-02T08:27:13.881 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-02T08:27:13.881 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-02T08:27:13.882 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-02T08:27:13.882 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-02T08:27:13.882 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-11-02T08:27:13.882 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-11-02T08:27:13.883 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-11-02T08:27:13.883 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-11-02T08:27:13.883 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-11-02T08:27:14.289 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:21 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:14.372 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-11-02T08:27:14.373 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:14.373 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:14.373 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:14.373 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-02T08:27:14.374 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-02T08:27:14.374 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-02T08:27:14.374 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-02T08:27:14.374 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-02T08:27:14.375 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-02T08:27:14.375 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-02T08:27:14.375 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-02T08:27:14.375 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-02T08:27:14.376 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-02T08:27:14.376 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-02T08:27:14.376 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:14.376 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:14.428 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:14.429 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:14.429 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:27:14.430 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:14.430 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:14.430 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 840 k 2023-11-02T08:27:14.430 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:27:14.431 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-02T08:27:14.431 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:14.431 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:14.431 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:14.431 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2023-11-02T08:27:14.431 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:14.432 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 919 k 2023-11-02T08:27:14.432 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.7 M 2023-11-02T08:27:14.432 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:14.553 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 640 kB/s | 79 kB 00:00 2023-11-02T08:27:14.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-rook 2023-11-02T08:27:14.928 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-fuse-18.2.0-1159.gb5a9486b.el9.x86_ 1.6 MB/s | 840 kB 00:00 2023-11-02T08:27:14.930 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:14.930 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.8 MB/s | 919 kB 00:00 2023-11-02T08:27:14.930 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:14.950 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:14.950 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:15.025 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:15.026 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:15.208 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:15.285 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:12 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:15.343 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-02T08:27:15.411 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/2 2023-11-02T08:27:15.430 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:15.432 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2023-11-02T08:27:15.432 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:27:15.433 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2023-11-02T08:27:15.433 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:15.433 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 47 k 2023-11-02T08:27:15.434 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:27:15.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-02T08:27:15.435 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-02T08:27:15.435 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-11-02T08:27:15.435 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-02T08:27:15.436 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-02T08:27:15.436 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-02T08:27:15.438 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-02T08:27:15.438 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-02T08:27:15.438 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-02T08:27:15.439 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-02T08:27:15.439 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-02T08:27:15.439 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-02T08:27:15.440 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:15.440 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:15.441 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2023-11-02T08:27:15.441 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2023-11-02T08:27:15.442 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:15.442 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 2.0 M 2023-11-02T08:27:15.443 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 26 M 2023-11-02T08:27:15.443 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:15.604 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 203 kB/s | 24 kB 00:00 2023-11-02T08:27:15.629 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2023-11-02T08:27:15.688 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2023-11-02T08:27:15.713 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2023-11-02T08:27:15.742 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-11-02T08:27:15.764 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2023-11-02T08:27:15.939 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): ceph-mgr-rook-18.2.0-1159.gb5a9486b.el9 104 kB/s | 47 kB 00:00 2023-11-02T08:27:16.006 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2023-11-02T08:27:16.098 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 45 kB/s | 15 kB 00:00 2023-11-02T08:27:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/2 2023-11-02T08:27:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/2 2023-11-02T08:27:16.231 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 447 kB/s | 59 kB 00:00 2023-11-02T08:27:16.306 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2023-11-02T08:27:16.373 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): python3-google-auth-2.23.3-1.el9.noarc 454 kB/s | 197 kB 00:00 2023-11-02T08:27:16.435 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-02T08:27:16.435 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:16.435 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-02T08:27:16.436 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:16.623 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2023-11-02T08:27:16.628 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:16.628 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2023-11-02T08:27:16.628 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:16.641 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:16.642 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:16.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-volume 2023-11-02T08:27:16.737 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:16.739 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:17.252 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:17.277 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:24 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:17.421 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:17.422 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2023-11-02T08:27:17.422 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:17.422 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 262 k 2023-11-02T08:27:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:17.423 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:17.423 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:17.423 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:27:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:17.424 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 262 k 2023-11-02T08:27:17.424 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.2 M 2023-11-02T08:27:17.424 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:17.428 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-02T08:27:17.525 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-02T08:27:17.589 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-02T08:27:17.625 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-02T08:27:17.667 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-02T08:27:17.760 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-02T08:27:17.854 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-volume-18.2.0-1159.gb5a9486b.el9.noarch.rp 608 kB/s | 262 kB 00:00 2023-11-02T08:27:17.854 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:17.855 INFO:teuthology.orchestra.run.smithi151.stdout:Total 606 kB/s | 262 kB 00:00 2023-11-02T08:27:17.855 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:17.860 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-11-02T08:27:17.862 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:17.863 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:17.894 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:17.894 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:17.918 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-02T08:27:18.106 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:18.165 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:27:18.180 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:27:18.180 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:27:18.181 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-02T08:27:18.181 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:18.641 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-02T08:27:18.690 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-02T08:27:18.724 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-02T08:27:18.770 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-02T08:27:18.794 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 13/13 2023-11-02T08:27:19.263 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:27:19.263 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:19.263 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:19.263 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:19.264 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:19.264 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:19.576 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 13/13 2023-11-02T08:27:19.576 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 1/13 2023-11-02T08:27:19.576 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-02T08:27:19.576 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-02T08:27:19.577 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-02T08:27:19.577 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-02T08:27:19.577 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-02T08:27:19.577 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-02T08:27:19.577 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-02T08:27:19.578 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-02T08:27:19.578 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-11-02T08:27:19.578 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-02T08:27:19.578 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-02T08:27:19.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel 2023-11-02T08:27:19.935 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-02T08:27:19.936 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:19.936 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:19.936 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:19.936 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-02T08:27:19.936 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-02T08:27:19.937 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-11-02T08:27:19.937 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-02T08:27:19.937 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-02T08:27:19.937 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-02T08:27:19.937 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-02T08:27:19.938 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-02T08:27:19.938 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-02T08:27:19.938 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-02T08:27:19.938 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-02T08:27:19.939 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-02T08:27:19.939 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:19.939 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:20.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-cephadm 2023-11-02T08:27:20.178 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:27 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:20.321 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:20.322 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:27:20.323 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:20.323 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 127 k 2023-11-02T08:27:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:20.323 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:20.324 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:20.324 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:27:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:20.324 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 127 k 2023-11-02T08:27:20.324 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 456 k 2023-11-02T08:27:20.325 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:20.621 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-18.2.0-1159.gb5a9486b.el9.x86_64 426 kB/s | 127 kB 00:00 2023-11-02T08:27:20.622 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:20.623 INFO:teuthology.orchestra.run.smithi151.stdout:Total 423 kB/s | 127 kB 00:00 2023-11-02T08:27:20.623 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:20.629 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:20.629 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:20.684 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:20.685 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:20.730 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:17 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:20.879 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:20.880 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2023-11-02T08:27:20.880 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:27:20.880 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2023-11-02T08:27:20.881 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:20.881 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 132 k 2023-11-02T08:27:20.881 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:27:20.881 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-02T08:27:20.881 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-02T08:27:20.881 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-02T08:27:20.882 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-02T08:27:20.882 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-02T08:27:20.882 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:20.882 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:20.882 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2023-11-02T08:27:20.883 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2023-11-02T08:27:20.883 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:20.883 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 6.8 M 2023-11-02T08:27:20.884 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 33 M 2023-11-02T08:27:20.884 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:21.219 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-11-02T08:27:21.344 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-11-02T08:27:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:21.603 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): ceph-mgr-cephadm-18.2.0-1159.gb5a9486b.e 246 kB/s | 132 kB 00:00 2023-11-02T08:27:21.708 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:21.709 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:21.709 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:21.709 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:21.709 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:21.709 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:21.803 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 133 kB/s | 58 kB 00:00 2023-11-02T08:27:21.970 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 731 kB/s | 548 kB 00:00 2023-11-02T08:27:21.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2 2023-11-02T08:27:22.037 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 118 kB/s | 51 kB 00:00 2023-11-02T08:27:22.040 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:22.041 INFO:teuthology.orchestra.run.smithi178.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2023-11-02T08:27:22.041 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:22.050 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:22.050 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:22.138 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:22.139 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:22.446 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:22.547 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:29 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:22.601 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-02T08:27:22.643 INFO:teuthology.orchestra.run.smithi151.stdout:Package libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:22.655 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-02T08:27:22.694 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:22.695 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:27:22.695 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:22.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs-devel 2023-11-02T08:27:23.176 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-02T08:27:23.258 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-02T08:27:23.315 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:30 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:23.317 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-02T08:27:23.351 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 6/6 2023-11-02T08:27:23.458 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:23.459 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:23.459 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:27:23.459 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:23.459 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:23.460 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 31 k 2023-11-02T08:27:23.460 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:23.460 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:23.460 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:23.460 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:27:23.461 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:23.461 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 31 k 2023-11-02T08:27:23.461 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 139 k 2023-11-02T08:27:23.461 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:23.796 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs-devel-18.2.0-1159.gb5a9486b.el9.x86_6 92 kB/s | 31 kB 00:00 2023-11-02T08:27:23.796 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:23.797 INFO:teuthology.orchestra.run.smithi151.stdout:Total 92 kB/s | 31 kB 00:00 2023-11-02T08:27:23.797 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:23.800 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:23.800 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:23.828 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:23.829 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:23.918 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:24.027 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:24.218 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 6/6 2023-11-02T08:27:24.218 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/6 2023-11-02T08:27:24.218 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-02T08:27:24.219 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-02T08:27:24.219 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-02T08:27:24.219 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-02T08:27:24.446 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:24.543 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-02T08:27:24.543 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:24.544 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:24.544 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:24.544 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-02T08:27:24.544 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-02T08:27:24.544 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-02T08:27:24.545 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-02T08:27:24.545 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-02T08:27:24.545 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:24.545 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:24.718 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:24.719 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:24.719 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:24.719 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:24.719 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:24.719 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:24.777 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse 2023-11-02T08:27:24.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2 2023-11-02T08:27:25.325 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:22 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:25.463 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:32 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:25.466 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:25.466 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:25.467 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:27:25.467 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:25.467 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:25.467 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 840 k 2023-11-02T08:27:25.467 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:27:25.468 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-02T08:27:25.468 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:25.468 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:25.468 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:25.468 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2023-11-02T08:27:25.469 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:25.469 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 919 k 2023-11-02T08:27:25.470 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.7 M 2023-11-02T08:27:25.470 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:25.555 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:25.588 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 656 kB/s | 79 kB 00:00 2023-11-02T08:27:25.605 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:25.606 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:27:25.607 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:25.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1 2023-11-02T08:27:25.829 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-fuse-18.2.0-1159.gb5a9486b.el9.x86_ 2.3 MB/s | 840 kB 00:00 2023-11-02T08:27:25.831 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:25.831 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.5 MB/s | 919 kB 00:00 2023-11-02T08:27:25.832 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:25.851 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:25.851 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:25.926 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:25.927 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:26.126 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:26.227 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:33 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:26.252 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-02T08:27:26.320 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:26.322 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/2 2023-11-02T08:27:26.370 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:26.371 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:27:26.371 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:26.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados 2023-11-02T08:27:26.997 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:33 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:27.025 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/2 2023-11-02T08:27:27.025 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/2 2023-11-02T08:27:27.091 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:27.141 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:27.141 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:27:27.142 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:27.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw 2023-11-02T08:27:27.335 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-02T08:27:27.335 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:27.336 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:27.336 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-02T08:27:27.336 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:27.336 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:27.600 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-volume 2023-11-02T08:27:27.761 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:34 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:27.854 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:27.905 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:27.905 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:27:27.906 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:27.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs 2023-11-02T08:27:28.157 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:25 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:28.301 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:28.302 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:28.303 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-02T08:27:28.303 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:28.303 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:28.303 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 262 k 2023-11-02T08:27:28.303 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:28.303 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:28.304 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:28.304 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:27:28.304 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:28.304 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 262 k 2023-11-02T08:27:28.304 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.2 M 2023-11-02T08:27:28.305 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:28.527 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-volume-18.2.0-1159.gb5a9486b.el9.noarch.rp 1.2 MB/s | 262 kB 00:00 2023-11-02T08:27:28.527 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:28.527 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-11-02T08:27:28.528 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:28.534 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:28.535 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:28.538 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:35 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:28.566 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:28.567 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:28.631 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:28.681 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:28.682 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:27:28.682 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:28.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd 2023-11-02T08:27:28.774 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:28.844 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:27:28.859 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:27:28.860 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:27:28.860 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-02T08:27:28.860 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:29.310 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:36 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:29.404 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:29.456 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:29.456 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:27:29.457 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:29.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse 2023-11-02T08:27:29.867 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:27:29.868 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:29.868 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:29.868 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:27:29.868 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:29.868 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:30.080 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:37 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:30.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel 2023-11-02T08:27:30.221 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:30.222 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:30.222 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2023-11-02T08:27:30.222 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:30.222 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:30.223 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 86 k 2023-11-02T08:27:30.223 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:30.223 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:30.223 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:30.223 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:27:30.223 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:30.224 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 86 k 2023-11-02T08:27:30.224 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 231 k 2023-11-02T08:27:30.224 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:30.475 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-18.2.0-1159.gb5a9486b.el9.x86_64.rpm 341 kB/s | 86 kB 00:00 2023-11-02T08:27:30.476 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:30.476 INFO:teuthology.orchestra.run.smithi151.stdout:Total 339 kB/s | 86 kB 00:00 2023-11-02T08:27:30.476 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:30.483 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:30.483 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:30.539 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:30.540 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:30.680 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:30.745 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:27 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:30.799 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:30.889 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:30.890 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:30.891 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:27:30.891 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:30.891 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:30.891 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 127 k 2023-11-02T08:27:30.891 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:30.892 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:30.892 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:30.892 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:27:30.892 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:30.892 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 127 k 2023-11-02T08:27:30.893 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 456 k 2023-11-02T08:27:30.893 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:31.122 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-18.2.0-1159.gb5a9486b.el9.x86_64 549 kB/s | 127 kB 00:00 2023-11-02T08:27:31.123 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:31.123 INFO:teuthology.orchestra.run.smithi178.stdout:Total 546 kB/s | 127 kB 00:00 2023-11-02T08:27:31.124 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:31.128 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:31.128 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:31.183 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:31.184 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:31.208 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:31.331 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:31.430 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:31.553 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:31.553 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:31.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror 2023-11-02T08:27:31.867 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:32.166 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:32.167 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:32.167 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:32.167 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:32.167 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:32.167 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:32.390 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:39 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:32.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2 2023-11-02T08:27:32.532 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:32.533 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:32.533 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:27:32.533 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:32.534 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:32.534 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 3.0 M 2023-11-02T08:27:32.534 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:32.534 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:32.534 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:32.535 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:27:32.535 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:32.535 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.0 M 2023-11-02T08:27:32.535 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 12 M 2023-11-02T08:27:32.537 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:32.983 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:29 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:33.078 INFO:teuthology.orchestra.run.smithi178.stdout:Package libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:33.130 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:33.131 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:27:33.131 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:33.197 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs-devel 2023-11-02T08:27:33.706 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-18.2.0-1159.gb5a9486b.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2023-11-02T08:27:33.707 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:33.707 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.5 MB/s | 3.0 MB 00:01 2023-11-02T08:27:33.708 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:33.718 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:33.718 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:33.752 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:30 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:33.791 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:33.792 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:33.893 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:33.894 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:33.894 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:27:33.894 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:33.895 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:33.895 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 31 k 2023-11-02T08:27:33.895 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:33.895 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:33.895 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:33.896 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:27:33.896 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:33.896 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 31 k 2023-11-02T08:27:33.896 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 139 k 2023-11-02T08:27:33.896 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:34.134 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs-devel-18.2.0-1159.gb5a9486b.el9.x86_6 130 kB/s | 31 kB 00:00 2023-11-02T08:27:34.134 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:34.135 INFO:teuthology.orchestra.run.smithi178.stdout:Total 129 kB/s | 31 kB 00:00 2023-11-02T08:27:34.135 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:34.138 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:34.138 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:34.161 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:34.162 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:34.205 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:34.255 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:34.266 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:34.290 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:34.291 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:27:34.291 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-02T08:27:34.291 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-02T08:27:34.291 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-02T08:27:34.292 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:34.398 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:34.787 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:35.050 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:35.051 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:35.051 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:35.051 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:35.052 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:35.052 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:35.243 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2 2023-11-02T08:27:35.416 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:35.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:35.417 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:35.417 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:35.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:35.418 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:35.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd 2023-11-02T08:27:35.796 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:32 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:35.887 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:35.937 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:35.938 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:27:35.938 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:35.998 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1 2023-11-02T08:27:36.226 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:43 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:36.368 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:36.370 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:36.370 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2023-11-02T08:27:36.370 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:36.371 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:36.371 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 170 k 2023-11-02T08:27:36.372 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:36.372 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:36.372 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:36.373 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:27:36.373 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:36.373 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 170 k 2023-11-02T08:27:36.374 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 490 k 2023-11-02T08:27:36.374 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:36.552 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:33 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:36.606 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-18.2.0-1159.gb5a9486b.el9.x86_64.rpm 720 kB/s | 170 kB 00:00 2023-11-02T08:27:36.607 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:36.607 INFO:teuthology.orchestra.run.smithi151.stdout:Total 716 kB/s | 170 kB 00:00 2023-11-02T08:27:36.608 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:36.615 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:36.615 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:36.643 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:36.670 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:36.670 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:36.692 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:36.694 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:27:36.694 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:36.754 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados 2023-11-02T08:27:36.812 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:37.303 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:34 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:37.395 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:37.446 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:37.447 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:27:37.447 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:37.503 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw 2023-11-02T08:27:37.707 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:37.707 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:37.707 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:37.708 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:37.708 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:37.708 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:37.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base-debuginfo 2023-11-02T08:27:38.048 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:35 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:38.140 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:38.189 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:38.190 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:27:38.190 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:38.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs 2023-11-02T08:27:38.551 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:45 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:27:38.696 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:27:38.697 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:38.697 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:27:38.697 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:38.697 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:27:38.698 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 74 M 2023-11-02T08:27:38.698 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2023-11-02T08:27:38.698 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 20 M 2023-11-02T08:27:38.698 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:38.698 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:27:38.699 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:27:38.699 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2023-11-02T08:27:38.699 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:38.700 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 94 M 2023-11-02T08:27:38.700 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 512 M 2023-11-02T08:27:38.700 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:27:38.806 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:35 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:38.897 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:38.946 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:38.947 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:27:38.947 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:39.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd 2023-11-02T08:27:39.558 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:36 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:39.650 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:27:39.700 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:39.700 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:27:39.701 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:39.765 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse 2023-11-02T08:27:40.313 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:37 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:40.452 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:40.453 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:40.454 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-02T08:27:40.454 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:40.454 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:40.454 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 86 k 2023-11-02T08:27:40.454 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:40.455 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:40.455 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:40.455 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:27:40.455 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:40.455 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 86 k 2023-11-02T08:27:40.456 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 231 k 2023-11-02T08:27:40.456 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:40.604 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): ceph-debuginfo-18.2.0-1159.gb5a9486b.el9 11 MB/s | 20 MB 00:01 2023-11-02T08:27:40.645 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-18.2.0-1159.gb5a9486b.el9.x86_64.rpm 452 kB/s | 86 kB 00:00 2023-11-02T08:27:40.645 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:40.646 INFO:teuthology.orchestra.run.smithi178.stdout:Total 449 kB/s | 86 kB 00:00 2023-11-02T08:27:40.646 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:40.652 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:40.652 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:40.706 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:40.706 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:40.856 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:40.967 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:41.369 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:41.673 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:41.674 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:41.674 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:41.674 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:41.674 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:41.674 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:41.921 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror 2023-11-02T08:27:42.479 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:39 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:42.626 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:42.628 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:42.628 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:27:42.628 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:42.629 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:42.629 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 3.0 M 2023-11-02T08:27:42.629 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:42.629 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:42.629 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:42.630 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:27:42.630 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:42.630 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.0 M 2023-11-02T08:27:42.630 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 12 M 2023-11-02T08:27:42.630 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:43.539 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-18.2.0-1159.gb5a9486b.el9.x86_64.rpm 3.3 MB/s | 3.0 MB 00:00 2023-11-02T08:27:43.539 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:43.540 INFO:teuthology.orchestra.run.smithi178.stdout:Total 3.3 MB/s | 3.0 MB 00:00 2023-11-02T08:27:43.540 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:43.550 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:43.550 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:43.627 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:43.628 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:43.646 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-base-debuginfo-18.2.0-1159.gb5a9486 15 MB/s | 74 MB 00:04 2023-11-02T08:27:43.648 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:43.648 INFO:teuthology.orchestra.run.smithi151.stdout:Total 19 MB/s | 94 MB 00:04 2023-11-02T08:27:43.649 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:27:43.653 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:27:43.653 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:27:44.037 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:44.114 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:44.138 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:44.138 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:27:44.138 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-02T08:27:44.139 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-02T08:27:44.139 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-02T08:27:44.139 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:44.256 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:27:44.256 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:27:45.258 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:45.258 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:45.259 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:45.259 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:45.259 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:45.259 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:45.493 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd 2023-11-02T08:27:46.057 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:43 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:46.198 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:46.198 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:46.199 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-02T08:27:46.199 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:46.199 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:46.200 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 170 k 2023-11-02T08:27:46.200 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:46.200 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:46.200 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:46.201 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:27:46.201 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:46.201 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 170 k 2023-11-02T08:27:46.201 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 490 k 2023-11-02T08:27:46.201 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:46.458 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-18.2.0-1159.gb5a9486b.el9.x86_64.rpm 655 kB/s | 170 kB 00:00 2023-11-02T08:27:46.459 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:46.459 INFO:teuthology.orchestra.run.smithi178.stdout:Total 652 kB/s | 170 kB 00:00 2023-11-02T08:27:46.460 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:46.467 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:46.467 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:46.522 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:46.523 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:46.672 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:46.773 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:47.172 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:27:47.201 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:47.521 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:27:47.522 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:47.522 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:27:47.522 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:47.522 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:47.523 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:27:47.814 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base-debuginfo 2023-11-02T08:27:48.380 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:45 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:27:48.522 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:27:48.523 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:48.524 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:27:48.524 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:48.524 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:27:48.524 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 74 M 2023-11-02T08:27:48.525 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-02T08:27:48.525 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 20 M 2023-11-02T08:27:48.525 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:48.525 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:27:48.525 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:27:48.526 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2023-11-02T08:27:48.526 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:27:48.526 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 94 M 2023-11-02T08:27:48.526 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 512 M 2023-11-02T08:27:48.527 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:27:52.110 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): ceph-base-debuginfo-18.2.0-1159.gb5a9486 21 MB/s | 74 MB 00:03 2023-11-02T08:27:53.302 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-debuginfo-18.2.0-1159.gb5a9486b.el9 4.3 MB/s | 20 MB 00:04 2023-11-02T08:27:53.304 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:53.304 INFO:teuthology.orchestra.run.smithi178.stdout:Total 20 MB/s | 94 MB 00:04 2023-11-02T08:27:53.305 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:27:53.308 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:27:53.309 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:27:53.920 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:27:53.920 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:27:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/2 2023-11-02T08:27:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 2/2 2023-11-02T08:27:56.823 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:27:58.812 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 2/2 2023-11-02T08:27:58.813 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/2 2023-11-02T08:27:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/2 2023-11-02T08:27:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:59.094 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:27:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:27:59.095 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:27:59.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-common-debuginfo 2023-11-02T08:27:59.897 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:02:06 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:28:00.042 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:28:00.043 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:28:00.043 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:28:00.043 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:28:00.043 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:28:00.043 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 404 M 2023-11-02T08:28:00.044 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:28:00.044 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:28:00.044 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:28:00.044 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:28:00.044 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:28:00.045 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 404 M 2023-11-02T08:28:00.045 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.7 G 2023-11-02T08:28:00.045 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:28:04.456 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/2 2023-11-02T08:28:04.527 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 2/2 2023-11-02T08:28:07.973 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 2/2 2023-11-02T08:28:07.974 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/2 2023-11-02T08:28:08.242 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/2 2023-11-02T08:28:08.243 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:28:08.243 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:28:08.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:08.243 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:08.244 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:28:08.244 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:28:08.458 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-common-debuginfo 2023-11-02T08:28:09.030 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:06 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:28:09.185 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:28:09.185 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:28:09.186 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:28:09.186 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:28:09.186 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:28:09.186 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 404 M 2023-11-02T08:28:09.187 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:28:09.187 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:28:09.187 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:28:09.187 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:28:09.187 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:28:09.188 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 404 M 2023-11-02T08:28:09.188 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.7 G 2023-11-02T08:28:09.188 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:28:26.668 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-common-debuginfo-18.2.0-1159.gb5a9486b.el9 15 MB/s | 404 MB 00:26 2023-11-02T08:28:26.668 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:26.669 INFO:teuthology.orchestra.run.smithi151.stdout:Total 15 MB/s | 404 MB 00:26 2023-11-02T08:28:26.669 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:28:26.673 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:28:26.673 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:28:29.103 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:28:29.104 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:28:37.486 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-common-debuginfo-18.2.0-1159.gb5a9486b.el9 14 MB/s | 404 MB 00:28 2023-11-02T08:28:37.487 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:37.487 INFO:teuthology.orchestra.run.smithi178.stdout:Total 14 MB/s | 404 MB 00:28 2023-11-02T08:28:37.488 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:28:37.492 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:28:37.492 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:28:39.915 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:28:39.916 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:29:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:29:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:29:19.651 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:29:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:29:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:19.923 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:29:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:29:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:19.924 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:29:20.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-11-02T08:29:20.764 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:27 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:29:20.914 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:29:20.915 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2023-11-02T08:29:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:29:20.915 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2023-11-02T08:29:20.915 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:29:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 1.4 M 2023-11-02T08:29:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:20.916 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:29:20.916 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2023-11-02T08:29:20.916 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:29:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:20.917 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 1.4 M 2023-11-02T08:29:20.917 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 6.2 M 2023-11-02T08:29:20.918 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:29:21.218 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-debuginfo-18.2.0-11 4.7 MB/s | 1.4 MB 00:00 2023-11-02T08:29:21.218 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:29:21.219 INFO:teuthology.orchestra.run.smithi151.stdout:Total 4.7 MB/s | 1.4 MB 00:00 2023-11-02T08:29:21.219 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:29:21.222 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:29:21.223 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:29:21.265 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:29:21.265 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:29:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:29:21.635 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.0-1159. 1/1 2023-11-02T08:29:21.649 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:29:21.756 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:29:22.057 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.0-1159. 1/1 2023-11-02T08:29:22.323 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.0-1159. 1/1 2023-11-02T08:29:22.324 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:22.324 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:29:22.324 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:29:22.324 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:22.325 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:29:22.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw-debuginfo 2023-11-02T08:29:23.080 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:30 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:29:23.223 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:29:23.224 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:29:23.224 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:29:23.225 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:29:23.225 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:29:23.225 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 163 M 2023-11-02T08:29:23.225 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:23.225 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:29:23.226 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:29:23.226 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:29:23.226 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:23.226 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 163 M 2023-11-02T08:29:23.226 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 680 M 2023-11-02T08:29:23.227 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:29:29.825 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:29:30.094 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:29:30.094 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:29:30.095 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:29:30.095 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:29:30.095 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:29:30.095 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:29:30.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-11-02T08:29:30.860 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:27 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:29:30.872 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-radosgw-debuginfo-18.2.0-1159.gb5a9486b.el 21 MB/s | 163 MB 00:07 2023-11-02T08:29:30.873 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:29:30.874 INFO:teuthology.orchestra.run.smithi151.stdout:Total 21 MB/s | 163 MB 00:07 2023-11-02T08:29:30.874 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:29:30.878 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:29:30.878 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:29:31.003 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:29:31.004 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2023-11-02T08:29:31.005 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:29:31.005 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2023-11-02T08:29:31.005 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:29:31.005 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 1.4 M 2023-11-02T08:29:31.006 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:29:31.006 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:29:31.006 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2023-11-02T08:29:31.006 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:29:31.006 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:29:31.007 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.4 M 2023-11-02T08:29:31.007 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 6.2 M 2023-11-02T08:29:31.007 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:29:31.307 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-debuginfo-18.2.0-11 4.7 MB/s | 1.4 MB 00:00 2023-11-02T08:29:31.308 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:29:31.308 INFO:teuthology.orchestra.run.smithi178.stdout:Total 4.7 MB/s | 1.4 MB 00:00 2023-11-02T08:29:31.308 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:29:31.311 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:29:31.312 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:29:31.353 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:29:31.353 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:29:31.603 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:29:31.705 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.0-1159. 1/1 2023-11-02T08:29:31.881 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:29:31.881 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:29:32.088 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.0-1159. 1/1 2023-11-02T08:29:32.352 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.0-1159. 1/1 2023-11-02T08:29:32.353 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:29:32.353 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:29:32.353 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:29:32.353 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:29:32.353 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:29:32.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw-debuginfo 2023-11-02T08:29:33.082 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:30 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:29:33.224 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:29:33.225 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:29:33.225 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:29:33.225 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:29:33.226 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:29:33.226 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 163 M 2023-11-02T08:29:33.226 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:29:33.226 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:29:33.226 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:29:33.227 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:29:33.227 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:29:33.227 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 163 M 2023-11-02T08:29:33.228 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 680 M 2023-11-02T08:29:33.228 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:29:41.268 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-radosgw-debuginfo-18.2.0-1159.gb5a9486b.el 20 MB/s | 163 MB 00:08 2023-11-02T08:29:41.269 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:29:41.270 INFO:teuthology.orchestra.run.smithi178.stdout:Total 20 MB/s | 163 MB 00:08 2023-11-02T08:29:41.270 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:29:41.274 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:29:41.274 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:29:42.265 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:29:42.265 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:29:48.543 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:29:48.652 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x 1/1 2023-11-02T08:29:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x 1/1 2023-11-02T08:29:53.535 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x 1/1 2023-11-02T08:29:53.536 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:53.536 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:29:53.536 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:29:53.536 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:53.536 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:29:53.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test-debuginfo 2023-11-02T08:29:54.314 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:04:01 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:29:54.454 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:29:54.455 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:29:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:29:54.456 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:29:54.456 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:29:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 602 M 2023-11-02T08:29:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:54.456 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:29:54.457 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:29:54.457 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:29:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:29:54.457 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 602 M 2023-11-02T08:29:54.457 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.7 G 2023-11-02T08:29:54.458 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:29:58.829 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:29:59.023 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x 1/1 2023-11-02T08:30:03.398 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x 1/1 2023-11-02T08:30:03.655 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x 1/1 2023-11-02T08:30:03.656 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:30:03.656 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:30:03.656 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:30:03.656 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:30:03.656 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:30:03.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test-debuginfo 2023-11-02T08:30:04.399 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:01 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:30:04.541 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:30:04.541 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:30:04.542 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:30:04.542 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:30:04.542 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:30:04.542 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 602 M 2023-11-02T08:30:04.543 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:30:04.543 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:30:04.543 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:30:04.543 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:30:04.543 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:30:04.544 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 602 M 2023-11-02T08:30:04.544 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.7 G 2023-11-02T08:30:04.544 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:30:26.071 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-test-debuginfo-18.2.0-1159.gb5a9486b.el9.x 19 MB/s | 602 MB 00:31 2023-11-02T08:30:26.073 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:30:26.073 INFO:teuthology.orchestra.run.smithi151.stdout:Total 19 MB/s | 602 MB 00:31 2023-11-02T08:30:26.074 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:30:26.078 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:30:26.079 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:30:29.663 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:30:29.664 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:30:42.712 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-test-debuginfo-18.2.0-1159.gb5a9486b.el9.x 16 MB/s | 602 MB 00:38 2023-11-02T08:30:42.713 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:30:42.713 INFO:teuthology.orchestra.run.smithi178.stdout:Total 16 MB/s | 602 MB 00:38 2023-11-02T08:30:42.714 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:30:42.718 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:30:42.718 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:30:46.281 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:30:46.282 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:31:32.973 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:31:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:31:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:31:42.696 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:31:42.696 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:31:42.696 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:31:42.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:31:42.697 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:31:42.697 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:31:42.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base-debuginfo 2023-11-02T08:31:43.546 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:50 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:31:43.649 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:31:43.702 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:31:43.702 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2023-11-02T08:31:43.703 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:31:43.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-debuginfo 2023-11-02T08:31:44.322 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:51 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:31:44.464 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:31:44.465 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:31:44.465 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:31:44.465 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:31:44.465 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:31:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 19 M 2023-11-02T08:31:44.466 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:31:44.466 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:31:44.466 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:31:44.466 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:31:44.467 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:31:44.467 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 19 M 2023-11-02T08:31:44.467 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 101 M 2023-11-02T08:31:44.467 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:31:46.478 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-mgr-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 9.5 MB/s | 19 MB 00:02 2023-11-02T08:31:46.479 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:31:46.480 INFO:teuthology.orchestra.run.smithi151.stdout:Total 9.5 MB/s | 19 MB 00:02 2023-11-02T08:31:46.480 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:31:46.483 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:31:46.483 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:31:46.629 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:31:46.629 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:31:48.823 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:31:48.944 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:31:49.208 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:31:49.418 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:31:49.904 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:31:50.169 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:31:50.169 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:31:50.170 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:31:50.170 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:31:50.170 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:31:50.170 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:31:50.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mds-debuginfo 2023-11-02T08:31:50.904 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:05:57 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:31:51.045 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:31:51.046 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:31:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:31:51.046 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:31:51.046 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:31:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 48 M 2023-11-02T08:31:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:31:51.047 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:31:51.047 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:31:51.047 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:31:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:31:51.048 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 48 M 2023-11-02T08:31:51.048 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 203 M 2023-11-02T08:31:51.048 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:31:53.442 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-mds-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 20 MB/s | 48 MB 00:02 2023-11-02T08:31:53.443 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:31:53.443 INFO:teuthology.orchestra.run.smithi151.stdout:Total 20 MB/s | 48 MB 00:02 2023-11-02T08:31:53.444 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:31:53.447 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:31:53.447 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:31:53.767 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:31:53.768 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:31:58.295 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:31:58.574 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:31:58.575 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:31:58.575 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:31:58.575 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:31:58.575 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:31:58.575 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:31:58.744 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:31:58.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base-debuginfo 2023-11-02T08:31:58.841 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:31:59.379 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:05:56 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:31:59.471 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:31:59.521 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:31:59.522 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-02T08:31:59.522 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:31:59.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-debuginfo 2023-11-02T08:32:00.128 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:05:57 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:32:00.269 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:32:00.270 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:00.271 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:32:00.271 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:00.271 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:32:00.271 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 19 M 2023-11-02T08:32:00.271 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:00.272 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:32:00.272 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:00.272 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:32:00.272 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 19 M 2023-11-02T08:32:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 101 M 2023-11-02T08:32:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:32:00.447 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:00.733 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:00.733 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:32:00.733 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:32:00.734 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:32:00.734 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:32:00.734 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:32:00.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mon-debuginfo 2023-11-02T08:32:01.362 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-mgr-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 18 MB/s | 19 MB 00:01 2023-11-02T08:32:01.363 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:32:01.364 INFO:teuthology.orchestra.run.smithi178.stdout:Total 18 MB/s | 19 MB 00:01 2023-11-02T08:32:01.364 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:32:01.367 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:32:01.367 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:32:01.514 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:32:01.515 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:32:01.535 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:06:08 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:32:01.678 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:32:01.679 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:32:01.679 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:32:01.679 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:32:01.679 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:32:01.680 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 113 M 2023-11-02T08:32:01.680 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:32:01.680 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:32:01.680 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:32:01.680 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:32:01.680 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:32:01.681 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 113 M 2023-11-02T08:32:01.681 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 525 M 2023-11-02T08:32:01.681 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:32:03.692 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:32:03.786 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:04.746 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:05.024 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:05.024 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:05.024 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:32:05.025 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:32:05.025 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:05.025 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:32:05.178 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mds-debuginfo 2023-11-02T08:32:05.724 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:06:02 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:32:05.865 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:32:05.866 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:05.867 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:32:05.867 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:05.868 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:32:05.868 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 48 M 2023-11-02T08:32:05.868 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:05.868 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:32:05.869 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:05.869 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:32:05.869 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:05.869 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 48 M 2023-11-02T08:32:05.869 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 203 M 2023-11-02T08:32:05.870 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:32:08.373 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-mds-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 19 MB/s | 48 MB 00:02 2023-11-02T08:32:08.374 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:32:08.374 INFO:teuthology.orchestra.run.smithi178.stdout:Total 19 MB/s | 48 MB 00:02 2023-11-02T08:32:08.374 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:32:08.378 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:32:08.379 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:32:08.697 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:32:08.697 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:32:13.677 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:32:13.770 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:15.376 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:15.646 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:15.646 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:15.646 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:32:15.647 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:32:15.647 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:15.647 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:32:15.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mon-debuginfo 2023-11-02T08:32:16.357 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:06:13 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:32:16.500 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:32:16.501 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:16.501 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:32:16.501 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:16.501 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:32:16.502 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 113 M 2023-11-02T08:32:16.502 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:16.502 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:32:16.502 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:16.502 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:32:16.503 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:16.503 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 113 M 2023-11-02T08:32:16.503 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 525 M 2023-11-02T08:32:16.503 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:32:22.249 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-mon-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 20 MB/s | 113 MB 00:05 2023-11-02T08:32:22.250 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:32:22.250 INFO:teuthology.orchestra.run.smithi178.stdout:Total 20 MB/s | 113 MB 00:05 2023-11-02T08:32:22.251 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:32:22.254 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:32:22.255 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:32:22.833 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-mon-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 5.3 MB/s | 113 MB 00:21 2023-11-02T08:32:22.834 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:32:22.834 INFO:teuthology.orchestra.run.smithi151.stdout:Total 5.3 MB/s | 113 MB 00:21 2023-11-02T08:32:22.835 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:32:22.838 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:32:22.839 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:32:22.948 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:32:22.949 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:32:23.536 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:32:23.536 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:32:34.796 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:32:34.896 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:35.449 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:32:35.621 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:38.593 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:38.593 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:38.594 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:32:38.594 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:32:38.594 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:38.594 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:32:38.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-osd-debuginfo 2023-11-02T08:32:39.052 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:39.320 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:06:36 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:32:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:32:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:32:39.333 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:32:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:32:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:32:39.334 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:32:39.460 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:32:39.461 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:39.461 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:32:39.461 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:39.461 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:32:39.461 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 312 M 2023-11-02T08:32:39.462 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:39.462 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:32:39.462 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:32:39.462 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:32:39.462 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:32:39.463 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 312 M 2023-11-02T08:32:39.463 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.3 G 2023-11-02T08:32:39.463 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:32:39.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-osd-debuginfo 2023-11-02T08:32:40.077 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:06:47 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:32:40.217 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:32:40.218 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:32:40.219 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:32:40.219 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:32:40.219 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:32:40.219 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 312 M 2023-11-02T08:32:40.219 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:32:40.220 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:32:40.220 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:32:40.220 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:32:40.220 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:32:40.220 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 312 M 2023-11-02T08:32:40.221 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.3 G 2023-11-02T08:32:40.221 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:32:55.598 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-osd-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 19 MB/s | 312 MB 00:16 2023-11-02T08:32:55.599 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:32:55.599 INFO:teuthology.orchestra.run.smithi178.stdout:Total 19 MB/s | 312 MB 00:16 2023-11-02T08:32:55.600 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:32:55.603 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:32:55.603 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:33:04.332 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-osd-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 13 MB/s | 312 MB 00:24 2023-11-02T08:33:04.332 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:33:04.333 INFO:teuthology.orchestra.run.smithi151.stdout:Total 13 MB/s | 312 MB 00:24 2023-11-02T08:33:04.333 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:33:04.339 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:33:04.339 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:33:06.213 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:33:06.213 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:33:29.297 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:33:29.409 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:33:37.599 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:33:37.873 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:33:37.873 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:37.873 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:33:37.874 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:33:37.874 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:37.875 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:33:38.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse-debuginfo 2023-11-02T08:33:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:33:38.434 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:33:38.626 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:07:35 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:33:38.766 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:33:38.767 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:38.767 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:33:38.767 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:38.767 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:33:38.768 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 5.7 M 2023-11-02T08:33:38.768 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:38.768 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:33:38.768 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:38.769 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:33:38.769 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:38.769 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 5.7 M 2023-11-02T08:33:38.769 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 28 M 2023-11-02T08:33:38.769 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:33:40.745 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-fuse-debuginfo-18.2.0-1159.gb5a9486b.el9.x 2.9 MB/s | 5.7 MB 00:01 2023-11-02T08:33:40.746 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:33:40.746 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.9 MB/s | 5.7 MB 00:01 2023-11-02T08:33:40.746 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:33:40.750 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:33:40.750 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:33:40.820 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:33:40.820 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:33:41.540 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:33:41.642 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:33:42.192 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:33:42.463 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:33:42.464 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:42.464 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:33:42.464 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:33:42.464 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:42.465 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:33:42.626 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel-debuginfo 2023-11-02T08:33:43.169 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:07:40 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:33:43.311 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:33:43.311 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:43.311 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:33:43.311 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:43.312 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:33:43.312 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 108 k 2023-11-02T08:33:43.312 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:43.312 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:33:43.312 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:43.313 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:33:43.313 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:43.313 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 108 k 2023-11-02T08:33:43.313 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 440 k 2023-11-02T08:33:43.313 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:33:43.589 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-debuginfo-18.2.0-1159.gb5a9486b. 390 kB/s | 108 kB 00:00 2023-11-02T08:33:43.590 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:33:43.590 INFO:teuthology.orchestra.run.smithi178.stdout:Total 388 kB/s | 108 kB 00:00 2023-11-02T08:33:43.591 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:33:43.593 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:33:43.593 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:33:43.627 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:33:43.628 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:33:43.746 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:33:43.853 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:33:44.228 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:33:44.488 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:33:44.488 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:44.488 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:33:44.489 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:33:44.489 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:44.489 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:33:44.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2-debuginfo 2023-11-02T08:33:45.195 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:07:42 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:33:45.336 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:33:45.337 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:45.337 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:33:45.337 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:45.337 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:33:45.337 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 4.9 M 2023-11-02T08:33:45.338 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:45.338 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:33:45.339 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:45.339 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:33:45.339 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:45.339 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 4.9 M 2023-11-02T08:33:45.339 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 23 M 2023-11-02T08:33:45.339 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:33:46.275 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs2-debuginfo-18.2.0-1159.gb5a9486b.el9. 5.2 MB/s | 4.9 MB 00:00 2023-11-02T08:33:46.276 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:33:46.277 INFO:teuthology.orchestra.run.smithi178.stdout:Total 5.2 MB/s | 4.9 MB 00:00 2023-11-02T08:33:46.277 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:33:46.280 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:33:46.280 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:33:46.342 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:33:46.343 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:33:46.904 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:33:46.960 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:33:47.073 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:33:47.175 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:33:47.175 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:47.176 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:33:47.176 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:33:47.176 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:47.176 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:33:47.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse-debuginfo 2023-11-02T08:33:47.577 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:33:47.852 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:33:47.853 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:47.853 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:33:47.853 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:33:47.853 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:47.854 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:33:47.956 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:07:54 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:33:48.009 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2-debuginfo 2023-11-02T08:33:48.096 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:33:48.097 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:48.097 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:33:48.098 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:48.098 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:33:48.098 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 5.7 M 2023-11-02T08:33:48.098 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:48.099 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:33:48.099 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:48.099 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:33:48.099 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:48.099 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 5.7 M 2023-11-02T08:33:48.100 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 28 M 2023-11-02T08:33:48.100 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:33:48.558 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:07:45 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:33:48.697 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:33:48.699 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:48.699 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:33:48.699 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:48.699 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:33:48.699 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 44 M 2023-11-02T08:33:48.700 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:48.700 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:33:48.700 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:48.700 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:33:48.700 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:48.701 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 44 M 2023-11-02T08:33:48.701 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 238 M 2023-11-02T08:33:48.701 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:33:48.921 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-fuse-debuginfo-18.2.0-1159.gb5a9486b.el9.x 7.0 MB/s | 5.7 MB 00:00 2023-11-02T08:33:48.923 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:33:48.923 INFO:teuthology.orchestra.run.smithi151.stdout:Total 6.9 MB/s | 5.7 MB 00:00 2023-11-02T08:33:48.924 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:33:48.926 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:33:48.926 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:33:48.993 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:33:48.993 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:33:49.726 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:33:49.827 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:33:50.442 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:33:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:33:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:50.707 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:33:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:33:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:50.707 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:33:50.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel-debuginfo 2023-11-02T08:33:51.194 INFO:teuthology.orchestra.run.smithi178.stdout:librados2-debuginfo-18.2.0-1159.gb5a9486b.el9.x 18 MB/s | 44 MB 00:02 2023-11-02T08:33:51.196 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:33:51.196 INFO:teuthology.orchestra.run.smithi178.stdout:Total 18 MB/s | 44 MB 00:02 2023-11-02T08:33:51.197 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:33:51.200 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:33:51.200 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:33:51.448 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:07:58 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:33:51.505 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:33:51.505 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:33:51.590 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:33:51.591 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:33:51.592 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:51.592 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:33:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 108 k 2023-11-02T08:33:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:51.592 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:33:51.593 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:51.593 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:33:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:51.593 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 108 k 2023-11-02T08:33:51.593 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 440 k 2023-11-02T08:33:51.593 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:33:51.816 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-debuginfo-18.2.0-1159.gb5a9486b. 485 kB/s | 108 kB 00:00 2023-11-02T08:33:51.816 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:33:51.817 INFO:teuthology.orchestra.run.smithi151.stdout:Total 482 kB/s | 108 kB 00:00 2023-11-02T08:33:51.817 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:33:51.820 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:33:51.821 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:33:51.854 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:33:51.854 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:33:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:33:52.072 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:33:52.456 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:33:52.723 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:33:52.723 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:52.723 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:33:52.724 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:33:52.724 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:52.724 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:33:52.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2-debuginfo 2023-11-02T08:33:53.463 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:00 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:33:53.604 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:33:53.605 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:33:53.605 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:53.605 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:33:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 4.9 M 2023-11-02T08:33:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:53.606 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:33:53.606 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:53.607 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:33:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:53.607 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 4.9 M 2023-11-02T08:33:53.607 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 23 M 2023-11-02T08:33:53.607 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:33:54.060 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs2-debuginfo-18.2.0-1159.gb5a9486b.el9. 11 MB/s | 4.9 MB 00:00 2023-11-02T08:33:54.061 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:33:54.062 INFO:teuthology.orchestra.run.smithi151.stdout:Total 11 MB/s | 4.9 MB 00:00 2023-11-02T08:33:54.062 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:33:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:33:54.065 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:33:54.127 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:33:54.127 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:33:54.750 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:33:54.858 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:33:55.375 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:33:55.648 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:33:55.648 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:55.649 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:33:55.649 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:33:55.649 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:55.649 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:33:55.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2-debuginfo 2023-11-02T08:33:56.415 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:03 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:33:56.483 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:33:56.555 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:33:56.556 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:56.556 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:33:56.557 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:56.557 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:33:56.557 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 44 M 2023-11-02T08:33:56.557 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:56.557 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:33:56.558 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:33:56.558 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:33:56.558 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:33:56.558 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 44 M 2023-11-02T08:33:56.558 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 238 M 2023-11-02T08:33:56.559 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:33:56.575 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:33:58.338 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:33:58.610 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:33:58.611 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:58.611 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:33:58.611 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:33:58.611 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:58.611 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:33:58.713 INFO:teuthology.orchestra.run.smithi151.stdout:librados2-debuginfo-18.2.0-1159.gb5a9486b.el9.x 20 MB/s | 44 MB 00:02 2023-11-02T08:33:58.714 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:33:58.714 INFO:teuthology.orchestra.run.smithi151.stdout:Total 20 MB/s | 44 MB 00:02 2023-11-02T08:33:58.715 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:33:58.718 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:33:58.719 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:33:58.791 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1-debuginfo 2023-11-02T08:33:59.021 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:33:59.022 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:33:59.344 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:07:56 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:33:59.487 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:33:59.488 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:59.488 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:33:59.488 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:59.488 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:33:59.489 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 51 M 2023-11-02T08:33:59.489 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:59.489 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:33:59.489 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:33:59.489 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:33:59.490 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:33:59.490 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 51 M 2023-11-02T08:33:59.490 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 292 M 2023-11-02T08:33:59.490 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:34:01.938 INFO:teuthology.orchestra.run.smithi178.stdout:librbd1-debuginfo-18.2.0-1159.gb5a9486b.el9.x86 21 MB/s | 51 MB 00:02 2023-11-02T08:34:01.939 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:01.939 INFO:teuthology.orchestra.run.smithi178.stdout:Total 21 MB/s | 51 MB 00:02 2023-11-02T08:34:01.940 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:34:01.944 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:34:01.944 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:34:02.288 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:34:02.289 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:34:03.955 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:34:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:34:05.954 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:34:06.230 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_ 1/1 2023-11-02T08:34:06.231 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:06.231 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:34:06.232 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:06.232 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:06.232 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:34:06.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1-debuginfo 2023-11-02T08:34:06.992 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:13 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:34:07.133 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:34:07.134 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:07.134 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:34:07.135 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:07.135 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:34:07.135 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 51 M 2023-11-02T08:34:07.135 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:07.135 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:34:07.136 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:07.136 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:34:07.136 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:07.136 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 51 M 2023-11-02T08:34:07.136 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 292 M 2023-11-02T08:34:07.137 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:34:08.102 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:34:08.203 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librbd1-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:09.607 INFO:teuthology.orchestra.run.smithi151.stdout:librbd1-debuginfo-18.2.0-1159.gb5a9486b.el9.x86 21 MB/s | 51 MB 00:02 2023-11-02T08:34:09.608 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:09.608 INFO:teuthology.orchestra.run.smithi151.stdout:Total 21 MB/s | 51 MB 00:02 2023-11-02T08:34:09.609 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:34:09.613 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:34:09.613 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:34:09.955 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:34:09.955 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:34:10.420 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:10.693 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:10.693 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:10.693 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:34:10.694 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:10.694 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:10.694 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:34:10.886 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs-debuginfo 2023-11-02T08:34:11.454 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:08 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:34:11.602 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:34:11.603 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:11.604 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:34:11.604 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:11.604 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:34:11.604 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 615 k 2023-11-02T08:34:11.604 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:11.605 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:34:11.605 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:11.605 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:34:11.605 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:11.605 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 615 k 2023-11-02T08:34:11.605 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.1 M 2023-11-02T08:34:11.606 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:34:11.907 INFO:teuthology.orchestra.run.smithi178.stdout:python3-cephfs-debuginfo-18.2.0-1159.gb5a9486b. 2.0 MB/s | 615 kB 00:00 2023-11-02T08:34:11.907 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:11.908 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.0 MB/s | 615 kB 00:00 2023-11-02T08:34:11.908 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:34:11.911 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:34:11.912 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:34:11.950 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:34:11.950 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:34:12.119 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:34:12.223 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:34:12.629 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:34:12.898 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:34:12.898 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:12.898 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:34:12.899 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:12.899 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:12.899 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:34:13.085 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados-debuginfo 2023-11-02T08:34:13.625 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:10 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:34:13.769 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:34:13.770 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:13.770 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:34:13.770 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:13.770 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:34:13.770 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 770 k 2023-11-02T08:34:13.771 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:13.771 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:34:13.771 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:13.771 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:34:13.771 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:13.772 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 770 k 2023-11-02T08:34:13.772 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.7 M 2023-11-02T08:34:13.772 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:34:14.845 INFO:teuthology.orchestra.run.smithi178.stdout:python3-rados-debuginfo-18.2.0-1159.gb5a9486b.e 715 kB/s | 770 kB 00:01 2023-11-02T08:34:14.846 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:14.846 INFO:teuthology.orchestra.run.smithi178.stdout:Total 715 kB/s | 770 kB 00:01 2023-11-02T08:34:14.846 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:34:14.848 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:34:14.848 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:34:14.888 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:34:14.889 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:34:15.070 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:34:15.184 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-debuginfo-2:18.2.0-1159.gb5a9486b.el9. 1/1 2023-11-02T08:34:15.605 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.0-1159.gb5a9486b.el9. 1/1 2023-11-02T08:34:15.781 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:34:15.865 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-debuginfo-2:18.2.0-1159.gb5a9486b.el9. 1/1 2023-11-02T08:34:15.865 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:15.865 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:34:15.865 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:15.866 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:15.866 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:34:15.880 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librbd1-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:16.039 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd-debuginfo 2023-11-02T08:34:16.586 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:13 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:34:16.726 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:34:16.727 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:16.727 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:34:16.728 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:16.728 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:34:16.728 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 1.1 M 2023-11-02T08:34:16.728 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:16.728 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:34:16.729 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:16.729 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:34:16.729 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:16.729 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.1 M 2023-11-02T08:34:16.729 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 4.3 M 2023-11-02T08:34:16.730 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:34:17.929 INFO:teuthology.orchestra.run.smithi178.stdout:python3-rbd-debuginfo-18.2.0-1159.gb5a9486b.el9 959 kB/s | 1.1 MB 00:01 2023-11-02T08:34:17.930 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:17.930 INFO:teuthology.orchestra.run.smithi178.stdout:Total 957 kB/s | 1.1 MB 00:01 2023-11-02T08:34:17.931 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:34:17.934 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:34:17.935 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:34:17.978 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:34:17.978 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:34:18.066 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:18.204 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:34:18.319 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:18.337 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:18.337 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:34:18.338 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:18.338 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:18.338 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:34:18.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs-debuginfo 2023-11-02T08:34:18.734 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:19.007 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:19.007 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:19.007 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:34:19.007 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:19.008 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:19.008 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:34:19.108 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:26 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:34:19.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw-debuginfo 2023-11-02T08:34:19.249 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:34:19.250 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:19.251 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:34:19.251 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:19.251 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:34:19.251 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 615 k 2023-11-02T08:34:19.251 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:19.252 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:34:19.252 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:19.252 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:34:19.252 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:19.252 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 615 k 2023-11-02T08:34:19.253 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.1 M 2023-11-02T08:34:19.253 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:34:19.757 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:16 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:34:19.898 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:34:19.899 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:19.899 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:34:19.899 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:19.899 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:34:19.899 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 338 k 2023-11-02T08:34:19.900 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:19.900 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:34:19.900 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:19.900 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:34:19.900 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:19.901 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 338 k 2023-11-02T08:34:19.901 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.0 M 2023-11-02T08:34:19.901 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:34:20.092 INFO:teuthology.orchestra.run.smithi151.stdout:python3-cephfs-debuginfo-18.2.0-1159.gb5a9486b. 731 kB/s | 615 kB 00:00 2023-11-02T08:34:20.093 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:20.093 INFO:teuthology.orchestra.run.smithi151.stdout:Total 730 kB/s | 615 kB 00:00 2023-11-02T08:34:20.094 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:34:20.097 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:34:20.097 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:34:20.136 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:34:20.137 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:34:20.266 INFO:teuthology.orchestra.run.smithi178.stdout:python3-rgw-debuginfo-18.2.0-1159.gb5a9486b.el9 922 kB/s | 338 kB 00:00 2023-11-02T08:34:20.267 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:20.267 INFO:teuthology.orchestra.run.smithi178.stdout:Total 918 kB/s | 338 kB 00:00 2023-11-02T08:34:20.267 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:34:20.270 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:34:20.270 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:34:20.300 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:34:20.306 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:34:20.306 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:34:20.416 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:34:20.442 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:34:20.548 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:34:20.919 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:21.128 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.0-1159.gb5a9486b.el9 1/1 2023-11-02T08:34:21.128 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:21.128 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:34:21.128 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:21.129 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:21.129 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:34:21.172 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:21.172 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:21.172 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:34:21.172 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:21.173 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:21.173 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:34:21.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados-debuginfo 2023-11-02T08:34:21.353 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse-debuginfo 2023-11-02T08:34:21.880 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:28 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:34:21.904 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:18 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:34:22.023 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:34:22.024 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:34:22.025 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:22.025 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:34:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 770 k 2023-11-02T08:34:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:22.027 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:34:22.027 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:22.027 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:34:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:22.027 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 770 k 2023-11-02T08:34:22.028 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.7 M 2023-11-02T08:34:22.028 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:34:22.045 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:34:22.046 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:22.047 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:34:22.047 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:22.047 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:34:22.047 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 507 k 2023-11-02T08:34:22.047 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:22.048 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:34:22.048 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:22.048 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:34:22.048 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:22.048 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 507 k 2023-11-02T08:34:22.049 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.5 M 2023-11-02T08:34:22.049 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:34:22.444 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 1.2 MB/s | 507 kB 00:00 2023-11-02T08:34:22.445 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:22.445 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.2 MB/s | 507 kB 00:00 2023-11-02T08:34:22.446 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:34:22.448 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:34:22.448 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:34:22.479 INFO:teuthology.orchestra.run.smithi151.stdout:python3-rados-debuginfo-18.2.0-1159.gb5a9486b.e 1.7 MB/s | 770 kB 00:00 2023-11-02T08:34:22.480 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:22.480 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.6 MB/s | 770 kB 00:00 2023-11-02T08:34:22.480 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:34:22.484 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:34:22.484 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:34:22.485 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:34:22.485 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:34:22.523 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:34:22.523 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:34:22.641 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:34:22.704 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:34:22.750 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:34:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-debuginfo-2:18.2.0-1159.gb5a9486b.el9. 1/1 2023-11-02T08:34:23.149 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:34:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.0-1159.gb5a9486b.el9. 1/1 2023-11-02T08:34:23.422 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:34:23.423 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:23.423 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:34:23.423 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:23.423 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:23.423 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:34:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-debuginfo-2:18.2.0-1159.gb5a9486b.el9. 1/1 2023-11-02T08:34:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:23.528 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:34:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:23.528 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:34:23.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror-debuginfo 2023-11-02T08:34:23.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd-debuginfo 2023-11-02T08:34:24.150 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:21 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:34:24.275 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:31 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:34:24.289 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:34:24.290 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:24.291 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:34:24.291 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:24.291 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:34:24.291 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 51 M 2023-11-02T08:34:24.291 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:24.292 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:34:24.292 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:24.292 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:34:24.292 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:24.292 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 51 M 2023-11-02T08:34:24.293 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 282 M 2023-11-02T08:34:24.293 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:34:24.415 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:34:24.416 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:24.416 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:34:24.416 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:24.416 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:34:24.417 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 1.1 M 2023-11-02T08:34:24.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:24.417 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:34:24.417 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:24.417 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:34:24.418 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:24.418 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 1.1 M 2023-11-02T08:34:24.419 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 4.3 M 2023-11-02T08:34:24.419 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:34:25.098 INFO:teuthology.orchestra.run.smithi151.stdout:python3-rbd-debuginfo-18.2.0-1159.gb5a9486b.el9 1.7 MB/s | 1.1 MB 00:00 2023-11-02T08:34:25.099 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:25.099 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.6 MB/s | 1.1 MB 00:00 2023-11-02T08:34:25.099 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:34:25.103 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:34:25.103 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:34:25.143 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:34:25.144 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:34:25.359 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:34:25.454 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:25.861 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:26.127 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:26.127 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:26.128 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:34:26.128 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:26.128 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:26.128 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:34:26.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw-debuginfo 2023-11-02T08:34:26.746 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-debuginfo-18.2.0-1159.gb5a9486b.el9. 21 MB/s | 51 MB 00:02 2023-11-02T08:34:26.747 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:26.748 INFO:teuthology.orchestra.run.smithi178.stdout:Total 21 MB/s | 51 MB 00:02 2023-11-02T08:34:26.748 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:34:26.752 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:34:26.752 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:34:26.900 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:33 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:34:27.042 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:34:27.043 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:34:27.043 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:27.043 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:34:27.043 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 338 k 2023-11-02T08:34:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:27.044 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:34:27.044 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:27.044 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:34:27.044 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:27.045 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 338 k 2023-11-02T08:34:27.045 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.0 M 2023-11-02T08:34:27.045 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:34:27.088 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:34:27.089 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:34:27.302 INFO:teuthology.orchestra.run.smithi151.stdout:python3-rgw-debuginfo-18.2.0-1159.gb5a9486b.el9 1.3 MB/s | 338 kB 00:00 2023-11-02T08:34:27.303 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:27.303 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.3 MB/s | 338 kB 00:00 2023-11-02T08:34:27.303 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:34:27.306 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:34:27.306 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:34:27.340 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:34:27.341 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:34:27.478 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:34:27.566 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:27.931 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x8 1/1 2023-11-02T08:34:28.219 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:28.219 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:34:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:28.220 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:28.220 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:34:28.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse-debuginfo 2023-11-02T08:34:28.974 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:35 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:34:29.116 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:34:29.117 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:29.117 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:34:29.117 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:29.117 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:34:29.117 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 507 k 2023-11-02T08:34:29.118 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:29.118 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:34:29.118 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:29.118 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:34:29.118 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:29.119 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 507 k 2023-11-02T08:34:29.119 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.5 M 2023-11-02T08:34:29.119 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:34:29.379 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-debuginfo-18.2.0-1159.gb5a9486b.el9.x8 1.9 MB/s | 507 kB 00:00 2023-11-02T08:34:29.380 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:29.380 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.9 MB/s | 507 kB 00:00 2023-11-02T08:34:29.381 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:34:29.383 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:34:29.384 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:34:29.420 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:34:29.420 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:34:29.576 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:34:29.678 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:34:30.070 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:34:30.334 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_6 1/1 2023-11-02T08:34:30.334 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:30.334 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:34:30.334 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:30.335 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:30.335 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:34:30.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror-debuginfo 2023-11-02T08:34:31.084 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:38 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:34:31.226 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:34:31.227 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:34:31.227 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:31.228 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:34:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 51 M 2023-11-02T08:34:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:31.228 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:34:31.228 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:31.228 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:34:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:31.229 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 51 M 2023-11-02T08:34:31.229 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 282 M 2023-11-02T08:34:31.229 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:34:32.824 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:34:32.927 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:34:33.640 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-debuginfo-18.2.0-1159.gb5a9486b.el9. 21 MB/s | 51 MB 00:02 2023-11-02T08:34:33.641 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:33.642 INFO:teuthology.orchestra.run.smithi151.stdout:Total 21 MB/s | 51 MB 00:02 2023-11-02T08:34:33.642 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:34:33.646 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:34:33.646 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:34:33.976 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:34:33.977 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:34:34.886 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:34:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:34:35.164 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:35.164 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:34:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:35.165 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:35.165 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:34:35.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd-debuginfo 2023-11-02T08:34:35.884 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:32 ago on Thu 02 Nov 2023 08:26:03 AM UTC. 2023-11-02T08:34:36.027 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-02T08:34:36.028 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:36.028 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-02T08:34:36.028 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:36.028 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-02T08:34:36.029 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 1.2 M 2023-11-02T08:34:36.029 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:36.029 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-02T08:34:36.029 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-02T08:34:36.029 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-02T08:34:36.030 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:36.030 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.2 M 2023-11-02T08:34:36.030 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 5.4 M 2023-11-02T08:34:36.030 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-02T08:34:36.681 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-debuginfo-18.2.0-1159.gb5a9486b.el9.x86 1.9 MB/s | 1.2 MB 00:00 2023-11-02T08:34:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.9 MB/s | 1.2 MB 00:00 2023-11-02T08:34:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-02T08:34:36.686 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-02T08:34:36.686 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-02T08:34:36.726 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-02T08:34:36.727 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-02T08:34:36.961 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-02T08:34:37.063 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:37.485 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:37.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:37.753 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:37.754 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-02T08:34:37.754 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:37.755 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:37.755 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-02T08:34:37.929 DEBUG:teuthology.parallel:result is None 2023-11-02T08:34:39.710 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:34:39.813 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:34:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:34:42.251 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86 1/1 2023-11-02T08:34:42.252 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:42.252 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:34:42.252 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:42.252 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:42.252 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:34:42.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd-debuginfo 2023-11-02T08:34:43.010 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:08:50 ago on Thu 02 Nov 2023 08:25:53 AM UTC. 2023-11-02T08:34:43.150 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2023-11-02T08:34:43.151 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2023-11-02T08:34:43.152 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:43.152 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2023-11-02T08:34:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-debuginfo x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 1.2 M 2023-11-02T08:34:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:43.152 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2023-11-02T08:34:43.153 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2023-11-02T08:34:43.153 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2023-11-02T08:34:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:43.153 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 1.2 M 2023-11-02T08:34:43.153 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 5.4 M 2023-11-02T08:34:43.154 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2023-11-02T08:34:43.594 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-debuginfo-18.2.0-1159.gb5a9486b.el9.x86 2.8 MB/s | 1.2 MB 00:00 2023-11-02T08:34:43.594 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:34:43.594 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.8 MB/s | 1.2 MB 00:00 2023-11-02T08:34:43.595 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2023-11-02T08:34:43.596 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2023-11-02T08:34:43.597 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2023-11-02T08:34:43.639 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2023-11-02T08:34:43.639 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2023-11-02T08:34:43.873 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2023-11-02T08:34:43.973 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:44.390 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:44.658 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:34:44.659 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:44.659 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2023-11-02T08:34:44.659 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-debuginfo-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:34:44.659 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:44.659 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2023-11-02T08:34:44.865 DEBUG:teuthology.parallel:result is None 2023-11-02T08:34:44.865 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:34:45.038 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-02T08:34:45.058 INFO:teuthology.orchestra.run.smithi151.stdout:18.2.0-1159.gb5a9486b.el9 2023-11-02T08:34:45.059 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-1159.gb5a9486b.el9 2023-11-02T08:34:45.060 INFO:teuthology.task.install:The correct ceph version 18.2.0-1159.gb5a9486b is installed. 2023-11-02T08:34:45.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:34:45.205 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-02T08:34:45.230 INFO:teuthology.orchestra.run.smithi178.stdout:18.2.0-1159.gb5a9486b.el9 2023-11-02T08:34:45.230 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-1159.gb5a9486b.el9 2023-11-02T08:34:45.231 INFO:teuthology.task.install:The correct ceph version 18.2.0-1159.gb5a9486b is installed. 2023-11-02T08:34:45.231 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-02T08:34:45.232 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:45.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-02T08:34:45.259 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:34:45.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-02T08:34:45.300 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-02T08:34:45.301 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:45.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2023-11-02T08:34:45.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-02T08:34:45.395 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:34:45.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2023-11-02T08:34:45.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-02T08:34:45.494 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-02T08:34:45.494 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:45.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-02T08:34:45.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-02T08:34:45.589 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:34:45.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-02T08:34:45.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-02T08:34:45.687 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-02T08:34:45.688 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:45.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2023-11-02T08:34:45.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-02T08:34:45.782 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:34:45.783 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2023-11-02T08:34:45.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-02T08:34:45.879 INFO:teuthology.run_tasks:Running task ceph... 2023-11-02T08:34:45.955 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-02T08:34:45.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /var/log/ceph 2023-11-02T08:34:45.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /var/log/ceph 2023-11-02T08:34:45.985 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-02T08:34:45.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-02T08:34:46.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-02T08:34:46.053 INFO:tasks.ceph:Creating extra log directories... 2023-11-02T08:34:46.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-02T08:34:46.091 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-02T08:34:46.132 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-02T08:34:46.133 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-02T08:34:46.133 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888', 'branch': 'wip-yuri6-testing-2023-11-01-0745-reef', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7443888', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2023-11-01-0745-reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2023-11-01-0745-reef', 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 49, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 525, 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri6-testing-2023-11-01-0745-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081', 'targets': {'smithi151.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKB+YlhA8QRvwhofhT9njZNwkD80E3wr4ljY0MpgRWCrpgX8xVCBUrUlMl/xOz6do5PWtzhVKyqQYgel4+hwS00=', 'smithi178.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWyM0BTTlZcSU06MdrOc/wX6OC7tSoQIDcXxB4KmJQ+YsjOrG0UckB44ObGExInMq2G8EWSWXFdPBEGaCx14rM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-01_21:37:41', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-11-02T08:34:46.134 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-02T08:34:46.158 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-02T08:34:46.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/run/ceph 2023-11-02T08:34:46.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/run/ceph 2023-11-02T08:34:46.243 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:46.243 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2023-11-02T08:34:46.294 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-02T08:34:46.294 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2023-11-02T08:34:46.348 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-02T08:34:46.349 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T08:34:46.349 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-11-02T08:34:46.349 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T08:34:46.349 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-11-02T08:34:46.350 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-11-02 08:27:34.519071057 +0000 2023-11-02T08:34:46.350 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-11-02 08:25:05.924809914 +0000 2023-11-02T08:34:46.350 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-11-02 08:25:05.924809914 +0000 2023-11-02T08:34:46.350 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2023-11-02 08:25:05.924809914 +0000 2023-11-02T08:34:46.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-02T08:34:46.414 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-11-02T08:34:46.414 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-11-02T08:34:46.414 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000188702 s, 2.7 MB/s 2023-11-02T08:34:46.421 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-02T08:34:46.476 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2023-11-02T08:34:46.533 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-02T08:34:46.533 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T08:34:46.533 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-11-02T08:34:46.534 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T08:34:46.534 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-11-02T08:34:46.534 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-11-02 08:27:34.519071057 +0000 2023-11-02T08:34:46.534 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-11-02 08:25:06.430803530 +0000 2023-11-02T08:34:46.534 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-11-02 08:25:06.430803530 +0000 2023-11-02T08:34:46.535 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2023-11-02 08:25:06.430803530 +0000 2023-11-02T08:34:46.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-02T08:34:46.598 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-11-02T08:34:46.598 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-11-02T08:34:46.599 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000170761 s, 3.0 MB/s 2023-11-02T08:34:46.600 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-02T08:34:46.655 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2023-11-02T08:34:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-02T08:34:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T08:34:46.710 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-11-02T08:34:46.711 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T08:34:46.711 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-11-02T08:34:46.711 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-11-02 08:27:34.520071044 +0000 2023-11-02T08:34:46.711 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-11-02 08:25:06.918797373 +0000 2023-11-02T08:34:46.711 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-11-02 08:25:06.918797373 +0000 2023-11-02T08:34:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2023-11-02 08:25:06.918797373 +0000 2023-11-02T08:34:46.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-02T08:34:46.775 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-11-02T08:34:46.775 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-11-02T08:34:46.775 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000188237 s, 2.7 MB/s 2023-11-02T08:34:46.777 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-02T08:34:46.833 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2023-11-02T08:34:46.888 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-02T08:34:46.888 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T08:34:46.888 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-11-02T08:34:46.889 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T08:34:46.889 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2023-11-02T08:34:46.889 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2023-11-02 08:27:34.520071044 +0000 2023-11-02T08:34:46.889 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2023-11-02 08:25:07.411791154 +0000 2023-11-02T08:34:46.889 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2023-11-02 08:25:07.411791154 +0000 2023-11-02T08:34:46.890 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2023-11-02 08:25:07.411791154 +0000 2023-11-02T08:34:46.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-02T08:34:46.953 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2023-11-02T08:34:46.953 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2023-11-02T08:34:46.954 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000195912 s, 2.6 MB/s 2023-11-02T08:34:46.955 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-02T08:34:47.011 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-02T08:34:47.012 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:34:47.012 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2023-11-02T08:34:47.029 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-02T08:34:47.030 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2023-11-02T08:34:47.085 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-02T08:34:47.086 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T08:34:47.086 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-11-02T08:34:47.086 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T08:34:47.087 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2023-11-02T08:34:47.087 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2023-11-02 08:27:44.393758757 +0000 2023-11-02T08:34:47.087 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2023-11-02 08:25:05.909003230 +0000 2023-11-02T08:34:47.087 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2023-11-02 08:25:05.909003230 +0000 2023-11-02T08:34:47.088 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2023-11-02 08:25:05.909003230 +0000 2023-11-02T08:34:47.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-02T08:34:47.152 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2023-11-02T08:34:47.152 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2023-11-02T08:34:47.153 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000108848 s, 4.7 MB/s 2023-11-02T08:34:47.154 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-02T08:34:47.210 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2023-11-02T08:34:47.265 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-02T08:34:47.266 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T08:34:47.266 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-11-02T08:34:47.266 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T08:34:47.266 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2023-11-02T08:34:47.267 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2023-11-02 08:27:44.393758757 +0000 2023-11-02T08:34:47.267 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2023-11-02 08:25:06.403993919 +0000 2023-11-02T08:34:47.267 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2023-11-02 08:25:06.403993919 +0000 2023-11-02T08:34:47.267 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2023-11-02 08:25:06.403993919 +0000 2023-11-02T08:34:47.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-02T08:34:47.331 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2023-11-02T08:34:47.332 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2023-11-02T08:34:47.332 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000102113 s, 5.0 MB/s 2023-11-02T08:34:47.333 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-02T08:34:47.390 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2023-11-02T08:34:47.445 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-02T08:34:47.446 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T08:34:47.446 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-11-02T08:34:47.446 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T08:34:47.446 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2023-11-02T08:34:47.446 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2023-11-02 08:27:44.393758757 +0000 2023-11-02T08:34:47.447 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2023-11-02 08:25:06.883984891 +0000 2023-11-02T08:34:47.447 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2023-11-02 08:25:06.883984891 +0000 2023-11-02T08:34:47.449 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2023-11-02 08:25:06.883984891 +0000 2023-11-02T08:34:47.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-02T08:34:47.510 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2023-11-02T08:34:47.511 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2023-11-02T08:34:47.511 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.00010482 s, 4.9 MB/s 2023-11-02T08:34:47.512 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-02T08:34:47.568 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2023-11-02T08:34:47.623 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-02T08:34:47.623 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-02T08:34:47.623 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-11-02T08:34:47.624 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-02T08:34:47.624 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2023-11-02T08:34:47.624 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2023-11-02 08:27:44.393758757 +0000 2023-11-02T08:34:47.624 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2023-11-02 08:25:07.365975824 +0000 2023-11-02T08:34:47.624 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2023-11-02 08:25:07.365975824 +0000 2023-11-02T08:34:47.625 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2023-11-02 08:25:07.365975824 +0000 2023-11-02T08:34:47.625 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-02T08:34:47.688 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2023-11-02T08:34:47.688 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2023-11-02T08:34:47.689 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 7.834e-05 s, 6.5 MB/s 2023-11-02T08:34:47.690 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-02T08:34:47.746 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-02T08:34:47.746 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-02T08:34:47.746 INFO:tasks.ceph:Generating config... 2023-11-02T08:34:47.748 INFO:tasks.ceph:[client] debug monc = 20 2023-11-02T08:34:47.748 INFO:tasks.ceph:[client] debug ms = 1 2023-11-02T08:34:47.748 INFO:tasks.ceph:[client] debug objecter = 20 2023-11-02T08:34:47.749 INFO:tasks.ceph:[client] debug rados = 20 2023-11-02T08:34:47.749 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-11-02T08:34:47.749 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-11-02T08:34:47.749 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-11-02T08:34:47.750 INFO:tasks.ceph:[global] ms type = async 2023-11-02T08:34:47.750 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-11-02T08:34:47.750 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-02T08:34:47.751 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-02T08:34:47.751 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-02T08:34:47.751 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-02T08:34:47.751 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-02T08:34:47.752 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-11-02T08:34:47.752 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-11-02T08:34:47.752 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-11-02T08:34:47.752 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-11-02T08:34:47.753 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-11-02T08:34:47.753 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-11-02T08:34:47.753 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-11-02T08:34:47.754 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-11-02T08:34:47.754 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-02T08:34:47.754 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-02T08:34:47.755 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-02T08:34:47.755 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-02T08:34:47.755 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-11-02T08:34:47.755 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-11-02T08:34:47.756 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-11-02T08:34:47.756 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-02T08:34:47.756 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-02T08:34:47.756 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-02T08:34:47.757 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-02T08:34:47.757 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-02T08:34:47.757 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-02T08:34:47.758 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-02T08:34:47.758 INFO:tasks.ceph:[osd] osd class default list = * 2023-11-02T08:34:47.758 INFO:tasks.ceph:[osd] osd class load list = * 2023-11-02T08:34:47.758 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-11-02T08:34:47.759 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-11-02T08:34:47.759 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-11-02T08:34:47.759 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-02T08:34:47.760 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-11-02T08:34:47.760 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-11-02T08:34:47.760 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-11-02T08:34:47.760 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-11-02T08:34:47.761 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-02T08:34:47.761 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-11-02T08:34:47.761 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-11-02T08:34:47.761 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-02T08:34:47.762 INFO:tasks.ceph:Setting up mon.a... 2023-11-02T08:34:47.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-02T08:34:47.813 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.keyring 2023-11-02T08:34:47.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-02T08:34:47.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-02T08:34:47.889 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.151'), ('mon.c', '[v2:172.21.15.151:3301,v1:172.21.15.151:6790]'), ('mon.b', '172.21.15.178')] 2023-11-02T08:34:47.889 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790],172.21.15.178', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-11-02T08:34:47.891 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:47.891 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-02T08:34:47.944 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.151 --addv c '[v2:172.21.15.151:3301,v1:172.21.15.151:6790]' --add b 172.21.15.178 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:34:48.028 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:34:48.029 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool: generated fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:34:48.029 INFO:teuthology.orchestra.run.smithi151.stdout:setting min_mon_release = pacific 2023-11-02T08:34:48.029 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 0 2023-11-02T08:34:48.030 INFO:teuthology.orchestra.run.smithi151.stdout:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:34:48.030 INFO:teuthology.orchestra.run.smithi151.stdout:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:34:48.030 INFO:teuthology.orchestra.run.smithi151.stdout:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:34:48.030 INFO:teuthology.orchestra.run.smithi151.stdout:min_mon_release 16 (pacific) 2023-11-02T08:34:48.031 INFO:teuthology.orchestra.run.smithi151.stdout:election_strategy: 1 2023-11-02T08:34:48.031 INFO:teuthology.orchestra.run.smithi151.stdout:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:34:48.031 INFO:teuthology.orchestra.run.smithi151.stdout:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:34:48.031 INFO:teuthology.orchestra.run.smithi151.stdout:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:34:48.031 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-02T08:34:48.033 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-02T08:34:48.048 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4d1a52ed-5f96-4808-8336-f35436fa00ba... 2023-11-02T08:34:48.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-02T08:34:48.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-02T08:34:48.137 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2023-11-02T08:34:48.137 INFO:teuthology.orchestra.run.smithi151.stdout: chdir = "" 2023-11-02T08:34:48.138 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-02T08:34:48.138 INFO:teuthology.orchestra.run.smithi151.stdout: auth supported = cephx 2023-11-02T08:34:48.138 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.138 INFO:teuthology.orchestra.run.smithi151.stdout: filestore xattr use omap = true 2023-11-02T08:34:48.138 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.139 INFO:teuthology.orchestra.run.smithi151.stdout: mon clock drift allowed = 1.000 2023-11-02T08:34:48.139 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.139 INFO:teuthology.orchestra.run.smithi151.stdout: osd crush chooseleaf type = 0 2023-11-02T08:34:48.139 INFO:teuthology.orchestra.run.smithi151.stdout: auth debug = true 2023-11-02T08:34:48.139 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.140 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on old message = true 2023-11-02T08:34:48.140 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on bug = true 2023-11-02T08:34:48.140 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.140 INFO:teuthology.orchestra.run.smithi151.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-02T08:34:48.140 INFO:teuthology.orchestra.run.smithi151.stdout: mon pg warn max object skew = 0 2023-11-02T08:34:48.140 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.141 INFO:teuthology.orchestra.run.smithi151.stdout: # disable pg_autoscaler by default for new pools 2023-11-02T08:34:48.141 INFO:teuthology.orchestra.run.smithi151.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-02T08:34:48.141 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.141 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default size = 2 2023-11-02T08:34:48.141 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.142 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow primary affinity = true 2023-11-02T08:34:48.142 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow pg remap = true 2023-11-02T08:34:48.142 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on legacy crush tunables = false 2023-11-02T08:34:48.142 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on crush straw calc version zero = false 2023-11-02T08:34:48.142 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on no sortbitwise = false 2023-11-02T08:34:48.142 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on osd down out interval zero = false 2023-11-02T08:34:48.143 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on too few osds = false 2023-11-02T08:34:48.143 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-02T08:34:48.143 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-02T08:34:48.143 INFO:teuthology.orchestra.run.smithi151.stdout: mon_allow_pool_size_one = true 2023-11-02T08:34:48.143 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.144 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-02T08:34:48.144 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.144 INFO:teuthology.orchestra.run.smithi151.stdout: osd default data pool replay window = 5 2023-11-02T08:34:48.144 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.144 INFO:teuthology.orchestra.run.smithi151.stdout: mon allow pool delete = true 2023-11-02T08:34:48.145 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.145 INFO:teuthology.orchestra.run.smithi151.stdout: mon cluster log file level = debug 2023-11-02T08:34:48.145 INFO:teuthology.orchestra.run.smithi151.stdout: debug asserts on shutdown = true 2023-11-02T08:34:48.145 INFO:teuthology.orchestra.run.smithi151.stdout: mon health detail to clog = false 2023-11-02T08:34:48.145 INFO:teuthology.orchestra.run.smithi151.stdout: mon host = "172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790],172.21.15.178" 2023-11-02T08:34:48.145 INFO:teuthology.orchestra.run.smithi151.stdout: mon client directed command retry = 5 2023-11-02T08:34:48.146 INFO:teuthology.orchestra.run.smithi151.stdout: mon election default strategy = 1 2023-11-02T08:34:48.146 INFO:teuthology.orchestra.run.smithi151.stdout: ms inject socket failures = 5000 2023-11-02T08:34:48.146 INFO:teuthology.orchestra.run.smithi151.stdout: ms type = async 2023-11-02T08:34:48.146 INFO:teuthology.orchestra.run.smithi151.stdout: osd heartbeat grace = 80 2023-11-02T08:34:48.146 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:34:48.147 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.147 INFO:teuthology.orchestra.run.smithi151.stdout:[osd] 2023-11-02T08:34:48.147 INFO:teuthology.orchestra.run.smithi151.stdout: osd journal size = 100 2023-11-02T08:34:48.147 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.147 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub load threshold = 5.0 2023-11-02T08:34:48.148 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub max interval = 600 2023-11-02T08:34:48.148 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock profile = high_recovery_ops 2023-11-02T08:34:48.148 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.148 INFO:teuthology.orchestra.run.smithi151.stdout: osd recover clone overlap = true 2023-11-02T08:34:48.148 INFO:teuthology.orchestra.run.smithi151.stdout: osd recovery max chunk = 1048576 2023-11-02T08:34:48.148 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.149 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug shutdown = true 2023-11-02T08:34:48.149 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug op order = true 2023-11-02T08:34:48.149 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify stray on activate = true 2023-11-02T08:34:48.149 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.149 INFO:teuthology.orchestra.run.smithi151.stdout: osd open classes on start = true 2023-11-02T08:34:48.150 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug pg log writeout = true 2023-11-02T08:34:48.150 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.150 INFO:teuthology.orchestra.run.smithi151.stdout: osd deep scrub update digest min age = 30 2023-11-02T08:34:48.150 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.150 INFO:teuthology.orchestra.run.smithi151.stdout: osd map max advance = 10 2023-11-02T08:34:48.151 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.151 INFO:teuthology.orchestra.run.smithi151.stdout: journal zero on create = true 2023-11-02T08:34:48.151 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.151 INFO:teuthology.orchestra.run.smithi151.stdout: filestore ondisk finisher threads = 3 2023-11-02T08:34:48.151 INFO:teuthology.orchestra.run.smithi151.stdout: filestore apply finisher threads = 3 2023-11-02T08:34:48.152 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.152 INFO:teuthology.orchestra.run.smithi151.stdout: bdev debug aio = true 2023-11-02T08:34:48.152 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug misdirected ops = true 2023-11-02T08:34:48.152 INFO:teuthology.orchestra.run.smithi151.stdout: bdev async discard = True 2023-11-02T08:34:48.152 INFO:teuthology.orchestra.run.smithi151.stdout: bdev enable discard = True 2023-11-02T08:34:48.153 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore block size = 96636764160 2023-11-02T08:34:48.153 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore fsck on mount = True 2023-11-02T08:34:48.153 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore zero block detection = True 2023-11-02T08:34:48.153 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluefs = 1 2023-11-02T08:34:48.153 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluestore = 1 2023-11-02T08:34:48.153 INFO:teuthology.orchestra.run.smithi151.stdout: debug monc = 20 2023-11-02T08:34:48.154 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-11-02T08:34:48.155 INFO:teuthology.orchestra.run.smithi151.stdout: debug osd = 20 2023-11-02T08:34:48.155 INFO:teuthology.orchestra.run.smithi151.stdout: debug rocksdb = 10 2023-11-02T08:34:48.155 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd backfillfull_ratio = 0.85 2023-11-02T08:34:48.155 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd full ratio = 0.9 2023-11-02T08:34:48.155 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd nearfull ratio = 0.8 2023-11-02T08:34:48.156 INFO:teuthology.orchestra.run.smithi151.stdout: osd class default list = * 2023-11-02T08:34:48.156 INFO:teuthology.orchestra.run.smithi151.stdout: osd class load list = * 2023-11-02T08:34:48.156 INFO:teuthology.orchestra.run.smithi151.stdout: osd client watch timeout = 120 2023-11-02T08:34:48.156 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify cached snaps = True 2023-11-02T08:34:48.156 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify missing on start = True 2023-11-02T08:34:48.157 INFO:teuthology.orchestra.run.smithi151.stdout: osd failsafe full ratio = 0.95 2023-11-02T08:34:48.157 INFO:teuthology.orchestra.run.smithi151.stdout: osd fast shutdown = False 2023-11-02T08:34:48.157 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock override recovery settings = True 2023-11-02T08:34:48.157 INFO:teuthology.orchestra.run.smithi151.stdout: osd memory target = 1610612736 2023-11-02T08:34:48.157 INFO:teuthology.orchestra.run.smithi151.stdout: osd objectstore = bluestore 2023-11-02T08:34:48.157 INFO:teuthology.orchestra.run.smithi151.stdout: osd op queue = debug_random 2023-11-02T08:34:48.158 INFO:teuthology.orchestra.run.smithi151.stdout: osd op queue cut off = debug_random 2023-11-02T08:34:48.158 INFO:teuthology.orchestra.run.smithi151.stdout: osd shutdown pgref assert = True 2023-11-02T08:34:48.158 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.158 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr] 2023-11-02T08:34:48.159 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-11-02T08:34:48.159 INFO:teuthology.orchestra.run.smithi151.stdout: debug mgr = 20 2023-11-02T08:34:48.159 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2023-11-02T08:34:48.159 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2023-11-02T08:34:48.159 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2023-11-02T08:34:48.159 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2023-11-02T08:34:48.160 INFO:teuthology.orchestra.run.smithi151.stdout: mgr/telemetry/nag = false 2023-11-02T08:34:48.160 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.160 INFO:teuthology.orchestra.run.smithi151.stdout:[mon] 2023-11-02T08:34:48.160 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-11-02T08:34:48.160 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2023-11-02T08:34:48.161 INFO:teuthology.orchestra.run.smithi151.stdout: debug paxos = 20 2023-11-02T08:34:48.161 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2023-11-02T08:34:48.161 INFO:teuthology.orchestra.run.smithi151.stdout: mon data avail warn = 5 2023-11-02T08:34:48.161 INFO:teuthology.orchestra.run.smithi151.stdout: mon mgr mkfs grace = 240 2023-11-02T08:34:48.161 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2023-11-02T08:34:48.161 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd reporter subtree level = osd 2023-11-02T08:34:48.162 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd prime pg temp = true 2023-11-02T08:34:48.162 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2023-11-02T08:34:48.162 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.162 INFO:teuthology.orchestra.run.smithi151.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-02T08:34:48.162 INFO:teuthology.orchestra.run.smithi151.stdout: auth mon ticket ttl = 660 # 11m 2023-11-02T08:34:48.163 INFO:teuthology.orchestra.run.smithi151.stdout: auth service ticket ttl = 240 # 4m 2023-11-02T08:34:48.163 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.163 INFO:teuthology.orchestra.run.smithi151.stdout: # don't complain about insecure global_id in the test suite 2023-11-02T08:34:48.163 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-02T08:34:48.163 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-02T08:34:48.164 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.164 INFO:teuthology.orchestra.run.smithi151.stdout: # 1m isn't quite enough 2023-11-02T08:34:48.164 INFO:teuthology.orchestra.run.smithi151.stdout: mon_down_mkfs_grace = 2m 2023-11-02T08:34:48.164 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.164 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_filestore_osds = false 2023-11-02T08:34:48.164 INFO:teuthology.orchestra.run.smithi151.stdout: mon min osdmap epochs = 50 2023-11-02T08:34:48.165 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd crush smoke test = False 2023-11-02T08:34:48.165 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune interval = 2 2023-11-02T08:34:48.165 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune min = 15 2023-11-02T08:34:48.165 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune txsize = 2 2023-11-02T08:34:48.165 INFO:teuthology.orchestra.run.smithi151.stdout: mon scrub interval = 300 2023-11-02T08:34:48.166 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on pool no app = False 2023-11-02T08:34:48.166 INFO:teuthology.orchestra.run.smithi151.stdout: paxos service trim min = 10 2023-11-02T08:34:48.166 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:34:48.166 INFO:teuthology.orchestra.run.smithi151.stdout:[client] 2023-11-02T08:34:48.166 INFO:teuthology.orchestra.run.smithi151.stdout: rgw cache enabled = true 2023-11-02T08:34:48.166 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable ops log = true 2023-11-02T08:34:48.167 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable usage log = true 2023-11-02T08:34:48.167 INFO:teuthology.orchestra.run.smithi151.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-02T08:34:48.167 INFO:teuthology.orchestra.run.smithi151.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-02T08:34:48.167 INFO:teuthology.orchestra.run.smithi151.stdout: debug monc = 20 2023-11-02T08:34:48.167 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2023-11-02T08:34:48.168 INFO:teuthology.orchestra.run.smithi151.stdout: debug objecter = 20 2023-11-02T08:34:48.168 INFO:teuthology.orchestra.run.smithi151.stdout: debug rados = 20 2023-11-02T08:34:48.168 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.a] 2023-11-02T08:34:48.168 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.c] 2023-11-02T08:34:48.168 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.b] 2023-11-02T08:34:48.170 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2023-11-02T08:34:48.170 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2023-11-02T08:34:48.170 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-02T08:34:48.171 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2023-11-02T08:34:48.171 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.171 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2023-11-02T08:34:48.171 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.171 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2023-11-02T08:34:48.172 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.172 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2023-11-02T08:34:48.172 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2023-11-02T08:34:48.172 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.172 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2023-11-02T08:34:48.173 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2023-11-02T08:34:48.173 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.173 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-02T08:34:48.173 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2023-11-02T08:34:48.173 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.173 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2023-11-02T08:34:48.174 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-02T08:34:48.174 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.174 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2023-11-02T08:34:48.174 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.174 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2023-11-02T08:34:48.175 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2023-11-02T08:34:48.175 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2023-11-02T08:34:48.175 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2023-11-02T08:34:48.175 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2023-11-02T08:34:48.175 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2023-11-02T08:34:48.175 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2023-11-02T08:34:48.176 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-02T08:34:48.176 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-02T08:34:48.176 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2023-11-02T08:34:48.176 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.176 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-02T08:34:48.177 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.177 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2023-11-02T08:34:48.177 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.177 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2023-11-02T08:34:48.177 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.177 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2023-11-02T08:34:48.178 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2023-11-02T08:34:48.178 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2023-11-02T08:34:48.178 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790],172.21.15.178" 2023-11-02T08:34:48.178 INFO:teuthology.orchestra.run.smithi178.stdout: mon client directed command retry = 5 2023-11-02T08:34:48.178 INFO:teuthology.orchestra.run.smithi178.stdout: mon election default strategy = 1 2023-11-02T08:34:48.179 INFO:teuthology.orchestra.run.smithi178.stdout: ms inject socket failures = 5000 2023-11-02T08:34:48.179 INFO:teuthology.orchestra.run.smithi178.stdout: ms type = async 2023-11-02T08:34:48.179 INFO:teuthology.orchestra.run.smithi178.stdout: osd heartbeat grace = 80 2023-11-02T08:34:48.179 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:34:48.179 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.180 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2023-11-02T08:34:48.180 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2023-11-02T08:34:48.180 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.180 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2023-11-02T08:34:48.180 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2023-11-02T08:34:48.180 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2023-11-02T08:34:48.181 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.181 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2023-11-02T08:34:48.181 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2023-11-02T08:34:48.181 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.181 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2023-11-02T08:34:48.182 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2023-11-02T08:34:48.182 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2023-11-02T08:34:48.182 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.182 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2023-11-02T08:34:48.182 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2023-11-02T08:34:48.182 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.183 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2023-11-02T08:34:48.183 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.183 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2023-11-02T08:34:48.183 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.183 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2023-11-02T08:34:48.184 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.184 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2023-11-02T08:34:48.184 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2023-11-02T08:34:48.184 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.184 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2023-11-02T08:34:48.184 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2023-11-02T08:34:48.185 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2023-11-02T08:34:48.185 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2023-11-02T08:34:48.185 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2023-11-02T08:34:48.185 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2023-11-02T08:34:48.185 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore zero block detection = True 2023-11-02T08:34:48.186 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1 2023-11-02T08:34:48.186 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1 2023-11-02T08:34:48.186 INFO:teuthology.orchestra.run.smithi178.stdout: debug monc = 20 2023-11-02T08:34:48.186 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2023-11-02T08:34:48.186 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2023-11-02T08:34:48.187 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 10 2023-11-02T08:34:48.187 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2023-11-02T08:34:48.187 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2023-11-02T08:34:48.187 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2023-11-02T08:34:48.187 INFO:teuthology.orchestra.run.smithi178.stdout: osd class default list = * 2023-11-02T08:34:48.187 INFO:teuthology.orchestra.run.smithi178.stdout: osd class load list = * 2023-11-02T08:34:48.188 INFO:teuthology.orchestra.run.smithi178.stdout: osd client watch timeout = 120 2023-11-02T08:34:48.188 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify cached snaps = True 2023-11-02T08:34:48.188 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify missing on start = True 2023-11-02T08:34:48.188 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2023-11-02T08:34:48.188 INFO:teuthology.orchestra.run.smithi178.stdout: osd fast shutdown = False 2023-11-02T08:34:48.189 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock override recovery settings = True 2023-11-02T08:34:48.189 INFO:teuthology.orchestra.run.smithi178.stdout: osd memory target = 1610612736 2023-11-02T08:34:48.189 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2023-11-02T08:34:48.189 INFO:teuthology.orchestra.run.smithi178.stdout: osd op queue = debug_random 2023-11-02T08:34:48.189 INFO:teuthology.orchestra.run.smithi178.stdout: osd op queue cut off = debug_random 2023-11-02T08:34:48.189 INFO:teuthology.orchestra.run.smithi178.stdout: osd shutdown pgref assert = True 2023-11-02T08:34:48.190 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.190 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2023-11-02T08:34:48.190 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2023-11-02T08:34:48.190 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2023-11-02T08:34:48.190 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2023-11-02T08:34:48.191 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2023-11-02T08:34:48.191 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2023-11-02T08:34:48.191 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2023-11-02T08:34:48.191 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2023-11-02T08:34:48.191 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.191 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2023-11-02T08:34:48.192 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2023-11-02T08:34:48.192 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2023-11-02T08:34:48.192 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2023-11-02T08:34:48.192 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2023-11-02T08:34:48.192 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2023-11-02T08:34:48.193 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2023-11-02T08:34:48.193 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2023-11-02T08:34:48.193 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2023-11-02T08:34:48.193 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2023-11-02T08:34:48.193 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2023-11-02T08:34:48.194 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.194 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-02T08:34:48.194 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2023-11-02T08:34:48.194 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2023-11-02T08:34:48.194 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.194 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2023-11-02T08:34:48.195 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-02T08:34:48.195 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-02T08:34:48.195 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.195 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2023-11-02T08:34:48.195 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2023-11-02T08:34:48.196 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.196 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2023-11-02T08:34:48.196 INFO:teuthology.orchestra.run.smithi178.stdout: mon min osdmap epochs = 50 2023-11-02T08:34:48.196 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd crush smoke test = False 2023-11-02T08:34:48.196 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune interval = 2 2023-11-02T08:34:48.196 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune min = 15 2023-11-02T08:34:48.197 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune txsize = 2 2023-11-02T08:34:48.197 INFO:teuthology.orchestra.run.smithi178.stdout: mon scrub interval = 300 2023-11-02T08:34:48.197 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on pool no app = False 2023-11-02T08:34:48.197 INFO:teuthology.orchestra.run.smithi178.stdout: paxos service trim min = 10 2023-11-02T08:34:48.197 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-02T08:34:48.198 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2023-11-02T08:34:48.198 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2023-11-02T08:34:48.198 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2023-11-02T08:34:48.198 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2023-11-02T08:34:48.198 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-02T08:34:48.198 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-02T08:34:48.199 INFO:teuthology.orchestra.run.smithi178.stdout: debug monc = 20 2023-11-02T08:34:48.199 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2023-11-02T08:34:48.199 INFO:teuthology.orchestra.run.smithi178.stdout: debug objecter = 20 2023-11-02T08:34:48.199 INFO:teuthology.orchestra.run.smithi178.stdout: debug rados = 20 2023-11-02T08:34:48.199 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2023-11-02T08:34:48.200 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2023-11-02T08:34:48.200 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2023-11-02T08:34:48.200 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-02T08:34:48.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-02T08:34:48.265 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-02T08:34:48.265 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:48.265 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-02T08:34:48.280 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:48.280 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-02T08:34:48.335 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T08:34:48.335 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:48.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-02T08:34:48.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-02T08:34:48.414 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:34:48.415 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:34:48.468 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T08:34:48.469 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:34:48.469 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-02T08:34:48.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-02T08:34:48.510 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:34:48.510 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:34:48.565 INFO:tasks.ceph:Setting up mon nodes... 2023-11-02T08:34:48.565 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-02T08:34:48.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-02T08:34:48.620 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-02T08:34:48.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-02T08:34:48.690 INFO:teuthology.orchestra.run.smithi178.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-02T08:34:48.692 INFO:tasks.ceph:Setting up mds nodes... 2023-11-02T08:34:48.693 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-02T08:34:48.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-02T08:34:48.735 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-02T08:34:48.750 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-02T08:34:48.792 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-02T08:34:48.808 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-02T08:34:48.808 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-02T08:34:48.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-02T08:34:48.835 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-02T08:34:48.836 INFO:tasks.ceph:role: osd.0 2023-11-02T08:34:48.836 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T08:34:48.836 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-02T08:34:48.909 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T08:34:48.909 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T08:34:48.909 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T08:34:48.910 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-02T08:34:48.910 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T08:34:48.910 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-11-02T08:34:48.910 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T08:34:48.910 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-02T08:34:48.911 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T08:34:48.911 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T08:34:49.418 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2023-11-02T08:34:49.420 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-11-02T08:34:49.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-02T08:34:49.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-02T08:34:49.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-02T08:34:49.663 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-02T08:34:49.663 INFO:tasks.ceph:role: osd.1 2023-11-02T08:34:49.663 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T08:34:49.664 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-02T08:34:49.739 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T08:34:49.739 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T08:34:49.740 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T08:34:49.740 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-02T08:34:49.740 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T08:34:49.740 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-11-02T08:34:49.741 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T08:34:49.741 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-02T08:34:49.741 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T08:34:49.741 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T08:34:50.190 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2023-11-02T08:34:50.192 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-11-02T08:34:50.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-02T08:34:50.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-02T08:34:50.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-02T08:34:50.387 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-02T08:34:50.387 INFO:tasks.ceph:role: osd.2 2023-11-02T08:34:50.387 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T08:34:50.388 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-02T08:34:50.462 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T08:34:50.463 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T08:34:50.463 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T08:34:50.463 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-02T08:34:50.463 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T08:34:50.464 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-11-02T08:34:50.464 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T08:34:50.464 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-02T08:34:50.464 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T08:34:50.464 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T08:34:50.959 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2023-11-02T08:34:50.961 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-11-02T08:34:50.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-02T08:34:51.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-02T08:34:51.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-02T08:34:51.211 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-02T08:34:51.211 INFO:tasks.ceph:role: osd.3 2023-11-02T08:34:51.212 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T08:34:51.212 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-02T08:34:51.285 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T08:34:51.285 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T08:34:51.286 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T08:34:51.286 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-02T08:34:51.286 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T08:34:51.286 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2023-11-02T08:34:51.286 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T08:34:51.287 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-02T08:34:51.287 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T08:34:51.287 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T08:34:51.804 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2023-11-02T08:34:51.806 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2023-11-02T08:34:51.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-02T08:34:51.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-02T08:34:51.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:34:52.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:52.072+0000 7ff1ee194500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-02T08:34:52.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:52.073+0000 7ff1ee194500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-02T08:34:52.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:52.073+0000 7ff1ee194500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-02T08:34:52.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:52.073+0000 7ff1ee194500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-02T08:34:52.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:52.073+0000 7ff1ee194500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-02T08:34:52.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:52.073+0000 7ff1ee194500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-02T08:34:54.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-02T08:34:54.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:34:55.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:55.042+0000 7f769589c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-02T08:34:55.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:55.042+0000 7f769589c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-02T08:34:55.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:55.042+0000 7f769589c500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-02T08:34:55.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:55.042+0000 7f769589c500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-02T08:34:55.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:55.042+0000 7f769589c500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-02T08:34:55.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:55.043+0000 7f769589c500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-02T08:34:57.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-02T08:34:57.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:34:58.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:58.023+0000 7fac73a49500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-02T08:34:58.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:58.023+0000 7fac73a49500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-02T08:34:58.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:58.023+0000 7fac73a49500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-02T08:34:58.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:58.023+0000 7fac73a49500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-02T08:34:58.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:58.023+0000 7fac73a49500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-02T08:34:58.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:34:58.024+0000 7fac73a49500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-02T08:35:00.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-02T08:35:00.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:35:01.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:01.000+0000 7f38c080f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-02T08:35:01.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:01.000+0000 7f38c080f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-02T08:35:01.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:01.000+0000 7f38c080f500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-02T08:35:01.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:01.000+0000 7f38c080f500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-02T08:35:01.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:01.000+0000 7f38c080f500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-02T08:35:01.005 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:01.001+0000 7f38c080f500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-02T08:35:03.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-02T08:35:03.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-02T08:35:03.922 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-02T08:35:03.923 INFO:tasks.ceph:role: osd.4 2023-11-02T08:35:03.923 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T08:35:03.923 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-02T08:35:03.992 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T08:35:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T08:35:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T08:35:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-02T08:35:03.993 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T08:35:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2023-11-02T08:35:03.994 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T08:35:03.994 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-02T08:35:03.994 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T08:35:03.994 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T08:35:04.024 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2023-11-02T08:35:04.027 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2023-11-02T08:35:04.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-02T08:35:04.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-02T08:35:04.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-02T08:35:04.242 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-02T08:35:04.243 INFO:tasks.ceph:role: osd.5 2023-11-02T08:35:04.243 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T08:35:04.243 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-02T08:35:04.309 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T08:35:04.309 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T08:35:04.310 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T08:35:04.310 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-02T08:35:04.310 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T08:35:04.310 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2023-11-02T08:35:04.311 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T08:35:04.311 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-02T08:35:04.311 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T08:35:04.312 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T08:35:04.341 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2023-11-02T08:35:04.342 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2023-11-02T08:35:04.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-02T08:35:04.413 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-02T08:35:04.485 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-02T08:35:04.552 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-02T08:35:04.552 INFO:tasks.ceph:role: osd.6 2023-11-02T08:35:04.553 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T08:35:04.553 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-02T08:35:04.618 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T08:35:04.619 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T08:35:04.619 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T08:35:04.619 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-02T08:35:04.619 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T08:35:04.619 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2023-11-02T08:35:04.620 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T08:35:04.620 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-02T08:35:04.620 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T08:35:04.620 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T08:35:04.652 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2023-11-02T08:35:04.653 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2023-11-02T08:35:04.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-02T08:35:04.723 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-02T08:35:04.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-02T08:35:04.862 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-02T08:35:04.862 INFO:tasks.ceph:role: osd.7 2023-11-02T08:35:04.862 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T08:35:04.863 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-02T08:35:04.928 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-02T08:35:04.929 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-02T08:35:04.929 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-02T08:35:04.929 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-02T08:35:04.929 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-02T08:35:04.929 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2023-11-02T08:35:04.930 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-02T08:35:04.930 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-02T08:35:04.930 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-02T08:35:04.930 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-02T08:35:04.961 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2023-11-02T08:35:04.963 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2023-11-02T08:35:04.963 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-02T08:35:05.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-02T08:35:05.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:35:05.201 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:05.197+0000 7f17c46ab500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-02T08:35:05.201 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:05.198+0000 7f17c46ab500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-02T08:35:05.202 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:05.198+0000 7f17c46ab500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-02T08:35:05.202 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:05.198+0000 7f17c46ab500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-02T08:35:05.202 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:05.198+0000 7f17c46ab500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-02T08:35:05.202 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:05.198+0000 7f17c46ab500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-02T08:35:08.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-02T08:35:08.097 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:35:08.187 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:08.183+0000 7f7233c14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-02T08:35:08.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:08.184+0000 7f7233c14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-02T08:35:08.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:08.184+0000 7f7233c14500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-02T08:35:08.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:08.184+0000 7f7233c14500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-02T08:35:08.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:08.184+0000 7f7233c14500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-02T08:35:08.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:08.184+0000 7f7233c14500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-02T08:35:11.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-02T08:35:11.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:35:11.157 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:11.153+0000 7f5d36e25500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-02T08:35:11.157 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:11.153+0000 7f5d36e25500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-02T08:35:11.158 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:11.153+0000 7f5d36e25500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-02T08:35:11.158 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:11.153+0000 7f5d36e25500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-02T08:35:11.158 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:11.153+0000 7f5d36e25500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-02T08:35:11.158 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:11.154+0000 7f5d36e25500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-02T08:35:14.000 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-02T08:35:14.031 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:35:14.122 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:14.118+0000 7f2c108cf500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-02T08:35:14.123 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:14.118+0000 7f2c108cf500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-02T08:35:14.123 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:14.119+0000 7f2c108cf500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-02T08:35:14.123 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:14.119+0000 7f2c108cf500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-02T08:35:14.123 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:14.119+0000 7f2c108cf500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-02T08:35:14.123 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:14.119+0000 7f2c108cf500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-02T08:35:16.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-02T08:35:16.989 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-02T08:35:16.989 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:16.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-02T08:35:17.022 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:17.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-02T08:35:17.091 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:17.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-02T08:35:17.159 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:17.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-02T08:35:17.228 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:17.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-02T08:35:17.295 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:17.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-02T08:35:17.327 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:17.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-02T08:35:17.397 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:17.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-02T08:35:17.466 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:17.467 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-02T08:35:17.535 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:17.535 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-02T08:35:17.604 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:17.604 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-02T08:35:17.621 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:17.621 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-02T08:35:17.660 INFO:tasks.ceph:Adding keys to all mons... 2023-11-02T08:35:17.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-02T08:35:17.663 DEBUG:teuthology.orchestra.run.smithi178:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-02T08:35:17.703 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.y] 2023-11-02T08:35:17.704 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQAoX0Nl8qvbJBAAfhTgVvSDvPYOvksbeDQyBQ== 2023-11-02T08:35:17.704 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.0] 2023-11-02T08:35:17.704 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQAsX0NlNDJoBBAAJ2QnbFqGKlXsf6LGlNPlAA== 2023-11-02T08:35:17.704 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.1] 2023-11-02T08:35:17.705 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQAvX0Nl5MCWAhAA6cBk1d+AmBSesxMl/xydkQ== 2023-11-02T08:35:17.705 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.2] 2023-11-02T08:35:17.705 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQAyX0NlLeJzARAA3xymQIn0jSeNf5D2cpjCdQ== 2023-11-02T08:35:17.705 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.3] 2023-11-02T08:35:17.706 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQA1X0NlvYgRABAA9StnRfrgAHfJ8m1ykedqiA== 2023-11-02T08:35:17.706 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.x] 2023-11-02T08:35:17.706 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQAoX0NlFOoAKRAA2plh5j8B+8hs9hP/pK06bQ== 2023-11-02T08:35:17.706 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.4] 2023-11-02T08:35:17.707 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQA5X0Nl4JLcCxAAJucQZJi0nbK1sbyxP0aoPQ== 2023-11-02T08:35:17.707 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.5] 2023-11-02T08:35:17.707 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQA8X0NlykoHCxAApt84ekVtTK47d/0hSTLzBw== 2023-11-02T08:35:17.708 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.6] 2023-11-02T08:35:17.708 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQA/X0NllUg4CRAAMj2FV6rtizkW1VWYHMI/+Q== 2023-11-02T08:35:17.708 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.7] 2023-11-02T08:35:17.708 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBCX0Nl/SYlBxAAM4eelDl3gfVya1PY1YudkA== 2023-11-02T08:35:17.711 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2023-11-02T08:35:17.712 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQAoX0NlqO+4KxAA3vtF3kKrlfk+Fyp6MWdGcg== 2023-11-02T08:35:17.712 INFO:teuthology.orchestra.run.smithi151.stdout:[client.1] 2023-11-02T08:35:17.712 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQAoX0NlFuMfLxAA2hPwo9v8ZHgqoPY6Rn1Rvg== 2023-11-02T08:35:17.726 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.y] 2023-11-02T08:35:17.727 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAoX0Nl8qvbJBAAfhTgVvSDvPYOvksbeDQyBQ== 2023-11-02T08:35:17.727 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.0] 2023-11-02T08:35:17.727 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAsX0NlNDJoBBAAJ2QnbFqGKlXsf6LGlNPlAA== 2023-11-02T08:35:17.728 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.1] 2023-11-02T08:35:17.728 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAvX0Nl5MCWAhAA6cBk1d+AmBSesxMl/xydkQ== 2023-11-02T08:35:17.728 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.2] 2023-11-02T08:35:17.728 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAyX0NlLeJzARAA3xymQIn0jSeNf5D2cpjCdQ== 2023-11-02T08:35:17.729 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.3] 2023-11-02T08:35:17.729 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQA1X0NlvYgRABAA9StnRfrgAHfJ8m1ykedqiA== 2023-11-02T08:35:17.729 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.x] 2023-11-02T08:35:17.729 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAoX0NlFOoAKRAA2plh5j8B+8hs9hP/pK06bQ== 2023-11-02T08:35:17.729 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.4] 2023-11-02T08:35:17.730 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQA5X0Nl4JLcCxAAJucQZJi0nbK1sbyxP0aoPQ== 2023-11-02T08:35:17.730 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.5] 2023-11-02T08:35:17.730 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQA8X0NlykoHCxAApt84ekVtTK47d/0hSTLzBw== 2023-11-02T08:35:17.730 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.6] 2023-11-02T08:35:17.731 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQA/X0NllUg4CRAAMj2FV6rtizkW1VWYHMI/+Q== 2023-11-02T08:35:17.731 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.7] 2023-11-02T08:35:17.731 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQBCX0Nl/SYlBxAAM4eelDl3gfVya1PY1YudkA== 2023-11-02T08:35:17.731 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2023-11-02T08:35:17.731 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAoX0NlqO+4KxAA3vtF3kKrlfk+Fyp6MWdGcg== 2023-11-02T08:35:17.732 INFO:teuthology.orchestra.run.smithi178.stdout:[client.1] 2023-11-02T08:35:17.732 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAoX0NlFuMfLxAA2hPwo9v8ZHgqoPY6Rn1Rvg== 2023-11-02T08:35:17.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-02T08:35:17.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-02T08:35:17.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:17.830 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:17.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:17.884 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:17.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:17.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:17.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:17.992 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:18.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-02T08:35:18.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-02T08:35:18.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:18.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:18.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:18.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:18.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:18.212 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:18.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:18.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-02T08:35:18.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-02T08:35:18.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-02T08:35:18.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-02T08:35:18.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-02T08:35:18.430 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-02T08:35:18.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-02T08:35:18.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-02T08:35:18.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-02T08:35:18.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-02T08:35:18.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-02T08:35:18.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-02T08:35:18.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-02T08:35:18.799 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-02T08:35:18.897 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-02T08:35:18.925 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:35:18.927 DEBUG:teuthology.orchestra.run.smithi178:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-02T08:35:18.979 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-02T08:35:18.980 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-11-02T08:35:18.981 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-02T08:35:18.981 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-11-02T08:35:18.984 INFO:tasks.ceph.mon.a:Started 2023-11-02T08:35:18.985 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-11-02T08:35:18.985 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-02T08:35:18.986 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-11-02T08:35:18.988 INFO:tasks.ceph.mon.c:Started 2023-11-02T08:35:18.989 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-11-02T08:35:18.989 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-02T08:35:18.990 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-11-02T08:35:19.023 INFO:tasks.ceph.mon.b:Started 2023-11-02T08:35:19.023 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-02T08:35:19.024 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-02T08:35:19.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-11-02T08:35:19.029 INFO:tasks.ceph.mgr.y:Started 2023-11-02T08:35:19.029 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-02T08:35:19.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-11-02T08:35:19.036 INFO:tasks.ceph.mgr.x:Started 2023-11-02T08:35:19.037 INFO:tasks.ceph:Setting crush tunables to default 2023-11-02T08:35:19.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd crush tunables default 2023-11-02T08:35:19.138 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.135+0000 7fb5aa5a9640 10 monclient: get_monmap_and_config 2023-11-02T08:35:19.139 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.135+0000 7fb5aa5a9640 10 monclient: build_initial_monmap 2023-11-02T08:35:19.139 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.135+0000 7fb5aa5a9640 10 monclient: monmap: 2023-11-02T08:35:19.139 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:35:19.139 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:19.140 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:35:19.136758+0000 2023-11-02T08:35:19.140 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:35:19.136758+0000 2023-11-02T08:35:19.140 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:19.140 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:19.140 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:19.141 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:19.141 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:19.141 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:19.141 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:19.141 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:19.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.136+0000 7fb5aa5a9640 1 Processor -- start 2023-11-02T08:35:19.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.136+0000 7fb5aa5a9640 1 -- start start 2023-11-02T08:35:19.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.136+0000 7fb5aa5a9640 10 monclient: init 2023-11-02T08:35:19.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:19.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-11-02T08:35:19.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 0x7fb5a4144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:19.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 10 monclient(hunting): picked mon.noname-c con 0x7fb5a40b4570 addr v2:172.21.15.178:3300/0 2023-11-02T08:35:19.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5a40b0ed0 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:19.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 0x7fb5a413e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:19.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 10 monclient(hunting): picked mon.noname-b con 0x7fb5a413e520 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:19.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a40a9780 con 0x7fb5a40b0ed0 2023-11-02T08:35:19.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:19.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a40b4950 con 0x7fb5a40b4570 2023-11-02T08:35:19.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:19.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a40aa1d0 con 0x7fb5a413e520 2023-11-02T08:35:19.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:19.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5aa5a9640 10 monclient(hunting): authenticate will time out at 1290.493045s 2023-11-02T08:35:19.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5a37fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb5a40b0ed0 legacy=0x7fb5a40a9070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T08:35:19.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5a8b46640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 msgr2=0x7fb5a413e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T08:35:19.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5a3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 msgr2=0x7fb5a4144d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T08:35:19.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5a8b46640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 0x7fb5a413e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:19.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.137+0000 7fb5a3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 0x7fb5a4144d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:19.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.338+0000 7fb5a8b46640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 msgr2=0x7fb5a413e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T08:35:19.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.338+0000 7fb5a37fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb5a40b0ed0 legacy=0x7fb5a40a9070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T08:35:19.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.338+0000 7fb5a8b46640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 0x7fb5a413e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T08:35:19.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.338+0000 7fb5a3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 msgr2=0x7fb5a4144d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T08:35:19.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.338+0000 7fb5a3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 0x7fb5a4144d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T08:35:19.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.739+0000 7fb5a8b46640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 msgr2=0x7fb5a413e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T08:35:19.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.739+0000 7fb5a37fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb5a40b0ed0 legacy=0x7fb5a40a9070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T08:35:19.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.739+0000 7fb5a8b46640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 0x7fb5a413e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T08:35:19.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.739+0000 7fb5a3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 msgr2=0x7fb5a4144d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T08:35:19.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:19.739+0000 7fb5a3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 0x7fb5a4144d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T08:35:20.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:20.540+0000 7fb5a8b46640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 msgr2=0x7fb5a413e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T08:35:20.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:20.540+0000 7fb5a37fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb5a40b0ed0 legacy=0x7fb5a40a9070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T08:35:20.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:20.540+0000 7fb5a8b46640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 0x7fb5a413e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T08:35:20.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:20.540+0000 7fb5a3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 msgr2=0x7fb5a4144d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T08:35:20.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:20.540+0000 7fb5a3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 0x7fb5a4144d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T08:35:22.140 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 10 monclient(hunting): tick 2023-11-02T08:35:22.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-02T08:35:22.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 1 monclient(hunting): continuing hunt 2023-11-02T08:35:22.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T08:35:22.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 msgr2=0x7fb5a413e9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:22.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5a413e520 0x7fb5a413e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:22.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 msgr2=0x7fb5a4144d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:22.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb5a40b4570 0x7fb5a4144d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:22.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb5a40b0ed0 legacy=0x7fb5a40a9070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:22.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 10 monclient: _add_conns ranks=[1,3] 2023-11-02T08:35:22.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5900036f0 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:22.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 0x7fb590009900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:22.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fb590007440 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:22.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a40aa4b0 con 0x7fb5900036f0 2023-11-02T08:35:22.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:22.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a40b4950 con 0x7fb590007440 2023-11-02T08:35:22.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a1ffb640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:22.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a37fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T08:35:22.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:22.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.137+0000 7fb5a3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb5900036f0 legacy=0x7fb590005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T08:35:22.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.338+0000 7fb5a37fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T08:35:22.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.338+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T08:35:22.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.338+0000 7fb5a3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb5900036f0 legacy=0x7fb590005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T08:35:22.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.739+0000 7fb5a37fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T08:35:22.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.739+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T08:35:22.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:22.739+0000 7fb5a3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb5900036f0 legacy=0x7fb590005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T08:35:23.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:23.540+0000 7fb5a37fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T08:35:23.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:23.540+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T08:35:23.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:23.540+0000 7fb5a3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb5900036f0 legacy=0x7fb590005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T08:35:25.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 10 monclient(hunting): tick 2023-11-02T08:35:25.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-02T08:35:25.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 1 monclient(hunting): continuing hunt 2023-11-02T08:35:25.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T08:35:25.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:25.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb590007440 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:25.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb5900036f0 legacy=0x7fb590005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:25.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a8b46640 1 -- reap_dead start 2023-11-02T08:35:25.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-11-02T08:35:25.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb590009fd0 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:25.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 0x7fb590009900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:25.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fb59000ab60 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:25.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb590005d20 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:25.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a40a9780 con 0x7fb590005d20 2023-11-02T08:35:25.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a40aa4b0 con 0x7fb590009fd0 2023-11-02T08:35:25.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:25.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a40b4950 con 0x7fb59000ab60 2023-11-02T08:35:25.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a1ffb640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:25.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a8b46640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb590005d20 legacy=0x7fb59000d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T08:35:25.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a37fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T08:35:25.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:25.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.137+0000 7fb5a3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb590009fd0 legacy=0x7fb59000a3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T08:35:25.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.338+0000 7fb5a8b46640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb590005d20 legacy=0x7fb59000d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T08:35:25.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.338+0000 7fb5a37fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T08:35:25.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.338+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T08:35:25.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.338+0000 7fb5a3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb590009fd0 legacy=0x7fb59000a3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T08:35:25.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.739+0000 7fb5a8b46640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb590005d20 legacy=0x7fb59000d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T08:35:25.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.739+0000 7fb5a37fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T08:35:25.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.739+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T08:35:25.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:25.739+0000 7fb5a3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb590009fd0 legacy=0x7fb59000a3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T08:35:26.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:26.540+0000 7fb5a8b46640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb590005d20 legacy=0x7fb59000d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T08:35:26.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:26.540+0000 7fb5a37fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T08:35:26.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:26.540+0000 7fb5a37fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T08:35:26.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:26.540+0000 7fb5a3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb590009fd0 legacy=0x7fb59000a3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T08:35:26.706 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:00:07.582 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:35:26.855 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:00:07.746 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:35:26.857 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:00:07.748 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:35:27.018 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:00:07.892 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:35:27.182 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:00:08.073 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:35:27.183 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:00:08.074 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:35:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 10 monclient(hunting): tick 2023-11-02T08:35:28.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-02T08:35:28.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 1 monclient(hunting): continuing hunt 2023-11-02T08:35:28.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T08:35:28.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 msgr2=0x7fb590009900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:28.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb59000ab60 0x7fb590009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:28.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fb590009fd0 legacy=0x7fb59000a3b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7fb590005d20 legacy=0x7fb59000d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 10 monclient: _add_conns ranks=[4,2] 2023-11-02T08:35:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:28.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fb59000e8b0 addr v2:172.21.15.178:3300/0 2023-11-02T08:35:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fb59000f330 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a40b4950 con 0x7fb59000e8b0 2023-11-02T08:35:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:28.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a40aa1d0 con 0x7fb59000f330 2023-11-02T08:35:28.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a1ffb640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:28.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a37fe640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 msgr2=0x7fb59000f740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T08:35:28.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:28.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 msgr2=0x7fb59000ecc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T08:35:28.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.137+0000 7fb5a3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:28.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.338+0000 7fb5a37fe640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 msgr2=0x7fb59000f740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T08:35:28.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.338+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T08:35:28.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.637+0000 7fb5a3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 msgr2=0x7fb59000ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-02T08:35:28.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.637+0000 7fb5a3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 msgr2=0x7fb59000ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:28.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.637+0000 7fb5a3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2023-11-02T08:35:28.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.637+0000 7fb5a3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T08:35:28.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.947+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:28.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.947+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:33780/0 (socket says 172.21.15.151:33780) 2023-11-02T08:35:28.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.947+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 learned_addr learned my addr 172.21.15.151:0/318434963 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:35:28.952 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.947+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request con 0x7fb59000f330 auth_method 0 2023-11-02T08:35:28.952 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.947+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:28.952 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.947+0000 7fb5a37fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:28.952 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.947+0000 7fb5a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:28.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.949+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 msgr2=0x7fb59000f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-02T08:35:28.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.949+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 msgr2=0x7fb59000f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:28.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.949+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:28.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:28.949+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T08:35:29.246 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.243+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:29.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.243+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request con 0x7fb59000e8b0 auth_method 0 2023-11-02T08:35:29.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.243+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:29.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.243+0000 7fb5a3fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:29.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.243+0000 7fb5a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:29.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.245+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 msgr2=0x7fb59000ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-02T08:35:29.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.245+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 msgr2=0x7fb59000ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:29.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.245+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:29.249 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.245+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T08:35:29.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.819+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:29.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.850+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request con 0x7fb59000f330 auth_method 0 2023-11-02T08:35:29.854 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.850+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:29.854 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.850+0000 7fb5a37fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:29.854 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.850+0000 7fb5a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:29.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.854+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 msgr2=0x7fb59000f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-02T08:35:29.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.854+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 msgr2=0x7fb59000f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:29.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.854+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:29.858 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:29.854+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T08:35:30.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:30.047+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:30.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:30.047+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request con 0x7fb59000e8b0 auth_method 0 2023-11-02T08:35:30.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:30.047+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:30.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:30.047+0000 7fb5a3fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:30.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:30.047+0000 7fb5a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:30.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:30.048+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 msgr2=0x7fb59000ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-02T08:35:30.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:30.048+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 msgr2=0x7fb59000ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:30.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:30.048+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:30.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:30.048+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T08:35:31.141 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): tick 2023-11-02T08:35:31.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): _check_auth_tickets 2023-11-02T08:35:31.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 monclient(hunting): continuing hunt 2023-11-02T08:35:31.142 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T08:35:31.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 msgr2=0x7fb59000f740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:31.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000f330 0x7fb59000f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:31.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 msgr2=0x7fb59000ecc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:31.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb59000e8b0 0x7fb59000ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:31.143 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 reap_dead start 2023-11-02T08:35:31.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-11-02T08:35:31.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:31.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fb590005d20 addr v2:172.21.15.178:3300/0 2023-11-02T08:35:31.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:31.144 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fb59000ab60 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:31.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:31.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fb590009880 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:31.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:31.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 -- 172.21.15.151:0/318434963 --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a40b4950 con 0x7fb590009880 2023-11-02T08:35:31.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:31.145 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 -- 172.21.15.151:0/318434963 --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a40aa1d0 con 0x7fb590005d20 2023-11-02T08:35:31.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:31.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 1 -- 172.21.15.151:0/318434963 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb590010340 con 0x7fb59000ab60 2023-11-02T08:35:31.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.137+0000 7fb5a1ffb640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:31.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:31.146 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:31.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request con 0x7fb590005d20 auth_method 0 2023-11-02T08:35:31.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:31.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a3fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:31.147 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:31.148 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request con 0x7fb59000ab60 auth_method 0 2023-11-02T08:35:31.148 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:31.148 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a37fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:31.148 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.138+0000 7fb5a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:31.149 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.140+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 msgr2=0x7fb59000f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-02T08:35:31.149 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.140+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 msgr2=0x7fb59000f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:31.149 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.140+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:31.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.140+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:31.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.144+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 msgr2=0x7fb59000ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-02T08:35:31.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.144+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 msgr2=0x7fb59000ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:31.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.144+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:31.150 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.144+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:31.178 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.174+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:31.179 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.177+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request con 0x7fb590009880 auth_method 0 2023-11-02T08:35:31.180 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.177+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:31.180 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.177+0000 7fb5a8b46640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:31.180 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.177+0000 7fb5a8b46640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:31.181 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.177+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 msgr2=0x7fb59000fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-02T08:35:31.181 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.177+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 msgr2=0x7fb59000fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:31.181 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.177+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:31.181 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.177+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:31.349 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.345+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:31.349 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.345+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request con 0x7fb590005d20 auth_method 0 2023-11-02T08:35:31.349 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.345+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:31.349 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.345+0000 7fb5a3fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:31.350 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.345+0000 7fb5a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:31.350 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.346+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 msgr2=0x7fb59000ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-02T08:35:31.351 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.346+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 msgr2=0x7fb59000ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:31.351 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.346+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:31.351 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.346+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T08:35:31.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.378+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:31.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.379+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request con 0x7fb590009880 auth_method 0 2023-11-02T08:35:31.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.379+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:31.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.379+0000 7fb5a8b46640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:31.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.379+0000 7fb5a8b46640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:31.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.381+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 msgr2=0x7fb59000fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-02T08:35:31.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.381+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 msgr2=0x7fb59000fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:31.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.381+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:31.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.381+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T08:35:31.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.391+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:31.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.392+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request con 0x7fb59000ab60 auth_method 0 2023-11-02T08:35:31.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.392+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:31.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.392+0000 7fb5a37fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:31.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.392+0000 7fb5a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:31.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.393+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 msgr2=0x7fb59000f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-02T08:35:31.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.393+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 msgr2=0x7fb59000f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:31.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.393+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:31.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.393+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T08:35:31.785 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.782+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:31.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.783+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request con 0x7fb590009880 auth_method 0 2023-11-02T08:35:31.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.783+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:31.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.783+0000 7fb5a8b46640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:31.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.783+0000 7fb5a8b46640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:31.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.784+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 msgr2=0x7fb59000fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-02T08:35:31.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.784+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 msgr2=0x7fb59000fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:31.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.784+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:31.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.784+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T08:35:31.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.804+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:31.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.805+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request con 0x7fb590005d20 auth_method 0 2023-11-02T08:35:31.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.805+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:31.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.805+0000 7fb5a3fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:31.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.805+0000 7fb5a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:31.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.808+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 msgr2=0x7fb59000ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-02T08:35:31.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.808+0000 7fb5a3fff640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 msgr2=0x7fb59000ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:31.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.808+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:31.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.808+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T08:35:31.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.865+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:31.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.872+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request con 0x7fb59000ab60 auth_method 0 2023-11-02T08:35:31.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.872+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:31.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.872+0000 7fb5a37fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:31.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.872+0000 7fb5a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:31.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.876+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 msgr2=0x7fb59000f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-02T08:35:31.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.876+0000 7fb5a37fe640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 msgr2=0x7fb59000f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:31.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.876+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:31.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:31.876+0000 7fb5a37fe640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T08:35:32.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.585+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.151:3300/0 conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:32.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.586+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request con 0x7fb590009880 auth_method 0 2023-11-02T08:35:32.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.586+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:32.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.586+0000 7fb5a8b46640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:32.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.586+0000 7fb5a8b46640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:32.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.586+0000 7fb5a8b46640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:32.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.586+0000 7fb5a8b46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:32.591 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.586+0000 7fb5a8b46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:32.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 10 monclient(hunting): handle_auth_done global_id 4101 payload 274 2023-11-02T08:35:32.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 msgr2=0x7fb59000f710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:32.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:32.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 msgr2=0x7fb59000ec90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T08:35:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 10 monclient: _finish_hunting 0 2023-11-02T08:35:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 1 monclient: found mon.noname-a 2023-11-02T08:35:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:32.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.588+0000 7fb5a8b46640 1 -- 172.21.15.151:0/318434963 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5900108e0 con 0x7fb590009880 2023-11-02T08:35:32.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.590+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 0x7fb59000fe00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb5a413e900 tx=0x7fb59c02c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97af479c8a522fdf server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:32.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 1 -- 172.21.15.151:0/318434963 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c032540 con 0x7fb590009880 2023-11-02T08:35:32.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:32.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:32.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 10 monclient: dump: 2023-11-02T08:35:32.598 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:35:32.598 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:32.600 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 10 monclient: _finish_auth 0 2023-11-02T08:35:32.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 10 monclient: _check_auth_tickets 2023-11-02T08:35:32.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:32.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 1 -- 172.21.15.151:0/318434963 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c02f030 con 0x7fb590009880 2023-11-02T08:35:32.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:32.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 1 -- 172.21.15.151:0/318434963 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c0326a0 con 0x7fb590009880 2023-11-02T08:35:32.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:32.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:32.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5a2ffd640 10 monclient: dump: 2023-11-02T08:35:32.602 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:35:32.602 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:32.604 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5aa5a9640 5 monclient: authenticate success, global_id 4101 2023-11-02T08:35:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5aa5a9640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:32.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5aa5a9640 4 set_mon_vals no callback set 2023-11-02T08:35:32.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5aa5a9640 10 monclient: shutdown 2023-11-02T08:35:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5aa5a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 msgr2=0x7fb59000fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.594+0000 7fb5aa5a9640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 0x7fb59000fe00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb5a413e900 tx=0x7fb59c02c5a0 comp rx=0 tx=0).stop 2023-11-02T08:35:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.595+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/318434963 shutdown_connections 2023-11-02T08:35:32.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.595+0000 7fb5aa5a9640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 0x7fb59000fe00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:32.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.595+0000 7fb5aa5a9640 1 --2- 172.21.15.151:0/318434963 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb59000f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:32.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.595+0000 7fb5aa5a9640 1 --2- 172.21.15.151:0/318434963 >> v2:172.21.15.178:3300/0 conn(0x7fb590005d20 0x7fb59000ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:32.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.595+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/318434963 >> 172.21.15.151:0/318434963 conn(0x7fb5a4065d50 msgr2=0x7fb5a4066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.595+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/318434963 shutdown_connections 2023-11-02T08:35:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.595+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/318434963 wait complete. 2023-11-02T08:35:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.595+0000 7fb5aa5a9640 10 monclient: build_initial_monmap 2023-11-02T08:35:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.595+0000 7fb5aa5a9640 10 monclient: monmap: 2023-11-02T08:35:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:35:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:35:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:35:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:32.614 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:32.614 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:32.614 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:32.614 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:32.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.596+0000 7fb5aa5a9640 1 librados: starting msgr at 2023-11-02T08:35:32.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.596+0000 7fb5aa5a9640 1 librados: starting objecter 2023-11-02T08:35:32.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.596+0000 7fb5aa5a9640 1 Processor -- start 2023-11-02T08:35:32.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.596+0000 7fb5aa5a9640 1 -- start start 2023-11-02T08:35:32.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.596+0000 7fb5aa5a9640 1 librados: setting wanted keys 2023-11-02T08:35:32.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.596+0000 7fb5aa5a9640 1 librados: calling monclient init 2023-11-02T08:35:32.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.596+0000 7fb5aa5a9640 10 monclient: init 2023-11-02T08:35:32.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:35:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb590005d20 0x7fb5a40abbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient(hunting): picked mon.noname-b con 0x7fb590005d20 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 0x7fb5a40aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient(hunting): picked mon.noname-a con 0x7fb590009880 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:32.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb5a41d9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:32.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient(hunting): picked mon.noname-c con 0x7fb59000ab60 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:32.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a41d9a50 con 0x7fb590009880 2023-11-02T08:35:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a4143030 con 0x7fb590005d20 2023-11-02T08:35:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a4143c70 con 0x7fb59000ab60 2023-11-02T08:35:32.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:32.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5aa5a9640 10 monclient(hunting): authenticate will time out at 1303.952948s 2023-11-02T08:35:32.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5a8b46640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb5a41d9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:32.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 0x7fb5a40aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:32.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5a3fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb590005d20 0x7fb5a40abbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:32.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 0x7fb5a40aaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36624/0 (socket says 172.21.15.151:36624) 2023-11-02T08:35:32.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5a37fe640 1 -- 172.21.15.151:0/2588327577 learned_addr learned my addr 172.21.15.151:0/2588327577 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:35:32.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request con 0x7fb590009880 auth_method 0 2023-11-02T08:35:32.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5a37fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.597+0000 7fb5a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request con 0x7fb59000ab60 auth_method 0 2023-11-02T08:35:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a8b46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a8b46640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:32.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a8b46640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:32.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request con 0x7fb590005d20 auth_method 0 2023-11-02T08:35:32.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:32.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a3fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:32.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a8b46640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a8b46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a8b46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.598+0000 7fb5a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2023-11-02T08:35:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 1 -- 172.21.15.151:0/2588327577 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 msgr2=0x7fb5a41d9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/2588327577 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb5a41d9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 1 -- 172.21.15.151:0/2588327577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 msgr2=0x7fb5a40aaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/2588327577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 0x7fb5a40aaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 10 monclient: _finish_hunting 0 2023-11-02T08:35:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 1 monclient: found mon.noname-b 2023-11-02T08:35:32.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:32.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:35:32.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a3fff640 1 -- 172.21.15.151:0/2588327577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c00a570 con 0x7fb590005d20 2023-11-02T08:35:32.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.601+0000 7fb5a8b46640 1 --2- 172.21.15.151:0/2588327577 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb5a41d9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-02T08:35:32.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.603+0000 7fb5a3fff640 1 --2- 172.21.15.151:0/2588327577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb590005d20 0x7fb5a40abbe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb594000c00 tx=0x7fb594005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:32.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.603+0000 7fb5a0ff9640 10 client.?.objecter ms_handle_connect 0x7fb590005d20 2023-11-02T08:35:32.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.603+0000 7fb5a0ff9640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:32.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 1 -- 172.21.15.151:0/2588327577 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594018660 con 0x7fb590005d20 2023-11-02T08:35:32.630 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:32.630 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:35:32.630 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 10 monclient: dump: 2023-11-02T08:35:32.630 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:35:32.630 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:32.632 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 10 monclient: _finish_auth 0 2023-11-02T08:35:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 10 monclient: _check_auth_tickets 2023-11-02T08:35:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 1 -- 172.21.15.151:0/2588327577 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb594011040 con 0x7fb590005d20 2023-11-02T08:35:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 1 -- 172.21.15.151:0/2588327577 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594018a40 con 0x7fb590005d20 2023-11-02T08:35:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5aa5a9640 5 monclient: authenticate success, global_id 4111 2023-11-02T08:35:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5aa5a9640 10 monclient: _renew_subs 2023-11-02T08:35:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5aa5a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/2588327577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a4141160 con 0x7fb590005d20 2023-11-02T08:35:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a27fc640 4 set_mon_vals no callback set 2023-11-02T08:35:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.607+0000 7fb5a0ff9640 10 monclient: dump: 2023-11-02T08:35:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:35:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:32.636 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:32.636 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:32.636 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:32.636 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:32.636 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:32.637 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.608+0000 7fb5aa5a9640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.608+0000 7fb5aa5a9640 10 monclient: _renew_subs 2023-11-02T08:35:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.608+0000 7fb5aa5a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.608+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/2588327577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a41da690 con 0x7fb590005d20 2023-11-02T08:35:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.608+0000 7fb5aa5a9640 1 librados: init done 2023-11-02T08:35:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.608+0000 7fb5aa5a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.608+0000 7fb5aa5a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.608+0000 7fb5aa5a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.608+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/2588327577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a41d98b0 con 0x7fb590005d20 2023-11-02T08:35:32.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.611+0000 7fb5a0ff9640 1 -- 172.21.15.151:0/2588327577 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb594006d50 con 0x7fb590005d20 2023-11-02T08:35:32.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.611+0000 7fb5a0ff9640 1 -- 172.21.15.151:0/2588327577 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb59402ad70 con 0x7fb590005d20 2023-11-02T08:35:32.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.611+0000 7fb5a0ff9640 10 client.4111.objecter ms_dispatch 0x7fb5a4000b90 osd_map(1..1 src has 1..1) v4 2023-11-02T08:35:32.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.611+0000 7fb5a0ff9640 3 client.4111.objecter handle_osd_map got epochs [1,1] > 0 2023-11-02T08:35:32.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.611+0000 7fb5a0ff9640 3 client.4111.objecter handle_osd_map decoding full epoch 1 2023-11-02T08:35:32.640 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.611+0000 7fb5a0ff9640 20 client.4111.objecter dump_active .. 0 homeless 2023-11-02T08:35:32.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.783+0000 7fb5a0ff9640 1 -- 172.21.15.151:0/2588327577 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb5a41d98b0 con 0x7fb590005d20 2023-11-02T08:35:32.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.783+0000 7fb5a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:32.787 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.783+0000 7fb5a0ff9640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:32.870 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:32.867+0000 7fb1fc810040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-02T08:35:32.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.868+0000 7fb5aa5a9640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-11-02T08:35:32.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.868+0000 7fb5aa5a9640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-11-02T08:35:32.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.868+0000 7fb5aa5a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:32.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:32.868+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/2588327577 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb5a41d9bc0 con 0x7fb590005d20 2023-11-02T08:35:32.899 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:32.896+0000 7f0836c1f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-02T08:35:32.922 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:32.920+0000 7fb1fc810040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-02T08:35:32.954 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:32.950+0000 7f0836c1f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-02T08:35:33.304 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.298+0000 7fb5a0ff9640 1 -- 172.21.15.151:0/2588327577 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb59402f090 con 0x7fb590005d20 2023-11-02T08:35:33.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.298+0000 7fb5a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-11-02T08:35:33.305 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.298+0000 7fb5a0ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-11-02T08:35:33.305 INFO:teuthology.orchestra.run.smithi151.stderr:adjusted tunables profile to default 2023-11-02T08:35:33.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 10 librados: watch_flush enter 2023-11-02T08:35:33.306 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 10 librados: watch_flush exit 2023-11-02T08:35:33.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 20 client.4111.objecter shutdown clearing up homeless session... 2023-11-02T08:35:33.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 10 client.4111.objecter successfully canceled tick 2023-11-02T08:35:33.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 10 monclient: shutdown 2023-11-02T08:35:33.307 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:33.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/2588327577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb590005d20 msgr2=0x7fb5a40abbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:33.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 --2- 172.21.15.151:0/2588327577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb590005d20 0x7fb5a40abbe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb594000c00 tx=0x7fb594005c00 comp rx=0 tx=0).stop 2023-11-02T08:35:33.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/2588327577 shutdown_connections 2023-11-02T08:35:33.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 --2- 172.21.15.151:0/2588327577 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb59000ab60 0x7fb5a41d9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:33.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 --2- 172.21.15.151:0/2588327577 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb590009880 0x7fb5a40aaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:33.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 --2- 172.21.15.151:0/2588327577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb590005d20 0x7fb5a40abbe0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:33.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/2588327577 >> 172.21.15.151:0/2588327577 conn(0x7fb5a4065d50 msgr2=0x7fb5a413c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:33.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/2588327577 shutdown_connections 2023-11-02T08:35:33.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 -- 172.21.15.151:0/2588327577 wait complete. 2023-11-02T08:35:33.311 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:33.299+0000 7fb5aa5a9640 1 librados: shutdown 2023-11-02T08:35:33.313 INFO:tasks.ceph:check_enable_crimson: False 2023-11-02T08:35:33.314 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-02T08:35:33.314 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:33.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-02T08:35:33.341 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:33.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-02T08:35:33.410 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:33.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-02T08:35:33.472 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.469+0000 7fb1fc810040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-02T08:35:33.477 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:35:33.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-02T08:35:33.508 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:33.505+0000 7f0836c1f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-02T08:35:33.523 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.521+0000 7fb1fc810040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-02T08:35:33.534 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:33.535 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-02T08:35:33.562 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:33.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-02T08:35:33.564 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:33.561+0000 7f0836c1f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-02T08:35:33.570 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.568+0000 7fb1fc810040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-02T08:35:33.597 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:33.597 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-02T08:35:33.612 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:33.609+0000 7f0836c1f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-02T08:35:33.627 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.625+0000 7fb1fc810040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-02T08:35:33.653 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-02T08:35:33.654 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-02T08:35:33.666 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:33.663+0000 7f0836c1f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-02T08:35:33.672 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.669+0000 7fb1fc810040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-02T08:35:33.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new fed323c8-4fdc-46c5-bf48-ba99f939a935 0 2023-11-02T08:35:33.708 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:33.705+0000 7f0836c1f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-02T08:35:33.777 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.775+0000 7fb1fc810040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-02T08:35:33.807 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:33.804+0000 7f0836c1f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-02T08:35:33.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.833+0000 7fbd60e51640 10 monclient: get_monmap_and_config 2023-11-02T08:35:33.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.833+0000 7fbd60e51640 10 monclient: build_initial_monmap 2023-11-02T08:35:33.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.833+0000 7fbd60e51640 10 monclient: monmap: 2023-11-02T08:35:33.837 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:33.837 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:33.837 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:35:33.833966+0000 2023-11-02T08:35:33.838 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:35:33.833966+0000 2023-11-02T08:35:33.838 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:33.838 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:33.838 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:33.838 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:33.839 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:33.839 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:33.839 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:33.839 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:33.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.834+0000 7fbd60e51640 1 Processor -- start 2023-11-02T08:35:33.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.834+0000 7fbd60e51640 1 -- start start 2023-11-02T08:35:33.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.834+0000 7fbd60e51640 10 monclient: init 2023-11-02T08:35:33.843 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:33.843 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-11-02T08:35:33.843 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd5c0b5b20 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:33.843 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 0x7fbd5c071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:33.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 10 monclient(hunting): picked mon.noname-c con 0x7fbd5c13f6b0 addr v2:172.21.15.178:3300/0 2023-11-02T08:35:33.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd5c13c490 0x7fbd5c13c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:33.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 10 monclient(hunting): picked mon.noname-b con 0x7fbd5c13c490 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:33.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5c142de0 con 0x7fbd5c0b5b20 2023-11-02T08:35:33.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:33.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c0b5f00 con 0x7fbd5c13f6b0 2023-11-02T08:35:33.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:33.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c071570 con 0x7fbd5c13c490 2023-11-02T08:35:33.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:33.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd60e51640 10 monclient(hunting): authenticate will time out at 1286.161261s 2023-11-02T08:35:33.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd53fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 0x7fbd5c071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:33.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.835+0000 7fbd5ad9d640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd5c13c490 0x7fbd5c13c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:33.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd5a59c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fbd5c0b5b20 0x7fbd5c143270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.178:49940/0 (socket says 172.21.15.178:49940) 2023-11-02T08:35:33.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd5a59c640 1 -- 172.21.15.178:0/4209678943 learned_addr learned my addr 172.21.15.178:0/4209678943 (peer_addr_for_me v1:172.21.15.178:0/0) 2023-11-02T08:35:33.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd53fff640 10 monclient(hunting): get_auth_request con 0x7fbd5c13f6b0 auth_method 0 2023-11-02T08:35:33.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:33.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd53fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:33.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd53fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:33.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd5ad9d640 10 monclient(hunting): get_auth_request con 0x7fbd5c13c490 auth_method 0 2023-11-02T08:35:33.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:33.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd5ad9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:33.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd5ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:33.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:33.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:33.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:33.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:33.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:33.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.836+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:33.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.837+0000 7fbd5a59c640 1 -- 172.21.15.178:0/4209678943 >> v1:172.21.15.151:6789/0 conn(0x7fbd5c0b5b20 legacy=0x7fbd5c143270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-02T08:35:33.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.837+0000 7fbd5a59c640 1 -- 172.21.15.178:0/4209678943 >> v1:172.21.15.151:6789/0 conn(0x7fbd5c0b5b20 legacy=0x7fbd5c143270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-02T08:35:33.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.837+0000 7fbd5a59c640 1 --1- 172.21.15.178:0/4209678943 >> v1:172.21.15.151:6789/0 conn(0x7fbd5c0b5b20 0x7fbd5c143270 :-1 s=OPENED pgs=8 cs=1 l=1).handle_message read tag failed 2023-11-02T08:35:33.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.837+0000 7fbd5a59c640 1 --1- 172.21.15.178:0/4209678943 >> v1:172.21.15.151:6789/0 conn(0x7fbd5c0b5b20 0x7fbd5c143270 :-1 s=OPENED pgs=8 cs=1 l=1).fault on lossy channel, failing 2023-11-02T08:35:33.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.837+0000 7fbd59d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.151:6789/0 2023-11-02T08:35:33.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.838+0000 7fbd53fff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2023-11-02T08:35:33.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.838+0000 7fbd53fff640 1 -- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd5c13c490 msgr2=0x7fbd5c13c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:33.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.838+0000 7fbd53fff640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd5c13c490 0x7fbd5c13c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:33.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.838+0000 7fbd53fff640 1 -- 172.21.15.178:0/4209678943 >> v1:172.21.15.151:6789/0 conn(0x7fbd5c0b5b20 legacy=0x7fbd5c143270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-02T08:35:33.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.838+0000 7fbd53fff640 10 monclient: _finish_hunting 0 2023-11-02T08:35:33.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.838+0000 7fbd53fff640 1 monclient: found mon.noname-c 2023-11-02T08:35:33.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.838+0000 7fbd53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:33.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.838+0000 7fbd53fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:35:33.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.838+0000 7fbd53fff640 1 -- 172.21.15.178:0/4209678943 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c0717e0 con 0x7fbd5c13f6b0 2023-11-02T08:35:33.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.840+0000 7fbd53fff640 1 -- 172.21.15.178:0/4209678943 >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 msgr2=0x7fbd5c071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-02T08:35:33.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.840+0000 7fbd53fff640 1 -- 172.21.15.178:0/4209678943 >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 msgr2=0x7fbd5c071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-02T08:35:33.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.840+0000 7fbd53fff640 1 --2- 172.21.15.178:0/4209678943 >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 0x7fbd5c071c00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fbd5c06d060 tx=0x7fbd4402b6e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-02T08:35:33.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:33.840+0000 7fbd53fff640 1 --2- 172.21.15.178:0/4209678943 >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 0x7fbd5c071c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T08:35:33.853 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.848+0000 7fb1fc810040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-02T08:35:33.878 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:33.875+0000 7f0836c1f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-02T08:35:33.896 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.893+0000 7fb1fc810040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-02T08:35:33.923 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:33.920+0000 7f0836c1f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-02T08:35:33.947 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.945+0000 7fb1fc810040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-02T08:35:33.977 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:33.973+0000 7f0836c1f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-02T08:35:33.995 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:33.993+0000 7fb1fc810040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-02T08:35:34.025 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:34.021+0000 7f0836c1f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-02T08:35:34.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.041+0000 7fbd53fff640 1 --2- 172.21.15.178:0/4209678943 >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 0x7fbd5c071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd53fff640 10 monclient: get_auth_request con 0x7fbd5c13f6b0 auth_method 0 2023-11-02T08:35:34.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd53fff640 0 --2- 172.21.15.178:0/4209678943 >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 0x7fbd5c071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-02T08:35:34.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd53fff640 1 --2- 172.21.15.178:0/4209678943 >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 0x7fbd5c071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 10 monclient: ms_handle_reset current mon v2:172.21.15.178:3300/0 2023-11-02T08:35:34.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:34.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 msgr2=0x7fbd5c071c00 unknown :-1 s=STATE_CLOSED l=0).mark_down 2023-11-02T08:35:34.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 1 --2- 172.21.15.178:0/4209678943 >> v2:172.21.15.178:3300/0 conn(0x7fbd5c13f6b0 0x7fbd5c071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 10 monclient: _add_conns ranks=[3,1] 2023-11-02T08:35:34.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 1 --2- 172.21.15.178:0/4209678943 >> v2:172.21.15.151:3300/0 conn(0x7fbd48003500 0x7fbd48005980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 10 monclient(hunting): picked mon.noname-a con 0x7fbd48003500 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:34.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd480062b0 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:34.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4800a220 con 0x7fbd480062b0 2023-11-02T08:35:34.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4800b3e0 con 0x7fbd48003500 2023-11-02T08:35:34.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:34.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd59d9b640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.178:3300/0 2023-11-02T08:35:34.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.042+0000 7fbd5a59c640 1 --2- 172.21.15.178:0/4209678943 >> v2:172.21.15.151:3300/0 conn(0x7fbd48003500 0x7fbd48005980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.043+0000 7fbd5a59c640 10 monclient(hunting): get_auth_request con 0x7fbd48003500 auth_method 0 2023-11-02T08:35:34.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.043+0000 7fbd5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:34.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.043+0000 7fbd5a59c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.043+0000 7fbd5a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:34.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.045+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482304828 0 0) 0x7fbd4800a220 con 0x7fbd480062b0 2023-11-02T08:35:34.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.045+0000 7fbd59d9b640 10 monclient(hunting): _negotiate 2023-11-02T08:35:34.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.045+0000 7fbd59d9b640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.045+0000 7fbd59d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:34.051 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.046+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 --> v1:172.21.15.178:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fbd4800a3b0 con 0x7fbd480062b0 2023-11-02T08:35:34.051 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:34.051 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:34.051 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-02T08:35:34.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1197303070 0 0) 0x7fbd4800a3b0 con 0x7fbd480062b0 2023-11-02T08:35:34.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 >> v2:172.21.15.151:3300/0 conn(0x7fbd48003500 msgr2=0x7fbd48005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:34.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 --2- 172.21.15.178:0/4209678943 >> v2:172.21.15.151:3300/0 conn(0x7fbd48003500 0x7fbd48005980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient: _finish_hunting 0 2023-11-02T08:35:34.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 monclient: found mon.noname-c-legacy 2023-11-02T08:35:34.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:34.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.178:6789/0 2023-11-02T08:35:34.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4800b610 con 0x7fbd480062b0 2023-11-02T08:35:34.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient: _finish_auth 0 2023-11-02T08:35:34.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient: _check_auth_tickets 2023-11-02T08:35:34.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:34.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 650167734 0 0) 0x7fbd44031940 con 0x7fbd480062b0 2023-11-02T08:35:34.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:34.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-11-02T08:35:34.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient: dump: 2023-11-02T08:35:34.057 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:34.057 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.057 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.057 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.057 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:34.058 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.058 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:34.058 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:34.058 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:34.058 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.058 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] but i'm connected to v1:172.21.15.178:6789/0, reconnecting 2023-11-02T08:35:34.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:34.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 >> v1:172.21.15.178:6789/0 conn(0x7fbd480062b0 legacy=0x7fbd48008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:35:34.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd5ad9d640 1 -- 172.21.15.178:0/4209678943 reap_dead start 2023-11-02T08:35:34.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800aa00 0x7fbd4800ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient(hunting): picked mon.b con 0x7fbd4800aa00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:34.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 0x7fbd4800d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient(hunting): picked mon.a con 0x7fbd48003970 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:34.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800dd90 0x7fbd48010220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient(hunting): picked mon.c con 0x7fbd4800dd90 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:34.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4800b3e0 con 0x7fbd48003970 2023-11-02T08:35:34.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48010ad0 con 0x7fbd4800aa00 2023-11-02T08:35:34.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48010c90 con 0x7fbd4800dd90 2023-11-02T08:35:34.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:34.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.049+0000 7fbd59d9b640 10 monclient(hunting): _finish_auth 0 2023-11-02T08:35:34.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.050+0000 7fbd5ad9d640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800dd90 0x7fbd48010220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.050+0000 7fbd53fff640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 0x7fbd4800d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.051+0000 7fbd5ad9d640 10 monclient(hunting): get_auth_request con 0x7fbd4800dd90 auth_method 0 2023-11-02T08:35:34.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.051+0000 7fbd5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:34.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.051+0000 7fbd5ad9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.051+0000 7fbd5ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:35:34.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.051+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:34.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.051+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:34.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.051+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-02T08:35:34.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5a59c640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800aa00 0x7fbd4800ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_done global_id 4114 payload 293 2023-11-02T08:35:34.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 1 -- 172.21.15.178:0/4209678943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800aa00 msgr2=0x7fbd4800ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:34.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800aa00 0x7fbd4800ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 1 -- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 msgr2=0x7fbd4800d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:34.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 0x7fbd4800d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 10 monclient: _finish_hunting 0 2023-11-02T08:35:34.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 1 monclient: found mon.c 2023-11-02T08:35:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:35:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 1 -- 172.21.15.178:0/4209678943 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4800a3b0 con 0x7fbd4800dd90 2023-11-02T08:35:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 10 monclient: _finish_auth 0 2023-11-02T08:35:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 10 monclient: _check_auth_tickets 2023-11-02T08:35:34.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd5ad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:34.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd60e51640 5 monclient: authenticate success, global_id 4114 2023-11-02T08:35:34.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.053+0000 7fbd60e51640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:35:34.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.056+0000 7fbd5ad9d640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800dd90 0x7fbd48010220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c070c30 tx=0x7fbd4800a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d3480fd0f30ca74 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:34.073 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54010040 con 0x7fbd4800dd90 2023-11-02T08:35:34.073 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:34.073 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:35:34.074 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 10 monclient: dump: 2023-11-02T08:35:34.074 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:34.074 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.075 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.075 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.075 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:34.076 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.076 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:34.076 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:34.076 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:34.077 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.077 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5400ee20 con 0x7fbd4800dd90 2023-11-02T08:35:34.077 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:34.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 1 -- 172.21.15.178:0/4209678943 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54014640 con 0x7fbd4800dd90 2023-11-02T08:35:34.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd60e51640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:34.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd60e51640 4 set_mon_vals no callback set 2023-11-02T08:35:34.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:34.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:35:34.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.069+0000 7fbd59d9b640 10 monclient: dump: 2023-11-02T08:35:34.079 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:34.080 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.080 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.080 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.080 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:34.080 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.080 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:34.081 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:34.081 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:34.081 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 10 monclient: shutdown 2023-11-02T08:35:34.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:34.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 1 -- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800dd90 msgr2=0x7fbd48010220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800dd90 0x7fbd48010220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c070c30 tx=0x7fbd4800a5b0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 1 -- 172.21.15.178:0/4209678943 shutdown_connections 2023-11-02T08:35:34.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800dd90 0x7fbd48010220 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 0x7fbd4800d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 1 --2- 172.21.15.178:0/4209678943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800aa00 0x7fbd4800ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 1 -- 172.21.15.178:0/4209678943 >> 172.21.15.178:0/4209678943 conn(0x7fbd5c065f90 msgr2=0x7fbd5c0b51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:34.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 1 -- 172.21.15.178:0/4209678943 shutdown_connections 2023-11-02T08:35:34.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.070+0000 7fbd60e51640 1 -- 172.21.15.178:0/4209678943 wait complete. 2023-11-02T08:35:34.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.071+0000 7fbd60e51640 10 monclient: build_initial_monmap 2023-11-02T08:35:34.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.071+0000 7fbd60e51640 10 monclient: monmap: 2023-11-02T08:35:34.086 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:34.087 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:34.087 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2023-11-02T08:35:34.087 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2023-11-02T08:35:34.087 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:34.088 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.088 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:34.088 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:34.089 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:34.089 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.071+0000 7fbd60e51640 1 librados: starting msgr at 2023-11-02T08:35:34.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.071+0000 7fbd60e51640 1 librados: starting objecter 2023-11-02T08:35:34.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.071+0000 7fbd60e51640 1 Processor -- start 2023-11-02T08:35:34.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.071+0000 7fbd60e51640 1 -- start start 2023-11-02T08:35:34.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.071+0000 7fbd60e51640 1 librados: setting wanted keys 2023-11-02T08:35:34.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.071+0000 7fbd60e51640 1 librados: calling monclient init 2023-11-02T08:35:34.092 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.071+0000 7fbd60e51640 10 monclient: init 2023-11-02T08:35:34.092 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:34.092 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:35:34.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 0x7fbd5c0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient(hunting): picked mon.noname-a con 0x7fbd48003970 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:34.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800aa00 0x7fbd5c1392f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient(hunting): picked mon.noname-c con 0x7fbd4800aa00 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:34.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800dd90 0x7fbd5c139830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient(hunting): picked mon.noname-b con 0x7fbd4800dd90 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:34.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0ab470 con 0x7fbd48003970 2023-11-02T08:35:34.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c139d70 con 0x7fbd4800dd90 2023-11-02T08:35:34.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c139ee0 con 0x7fbd4800aa00 2023-11-02T08:35:34.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:34.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.072+0000 7fbd60e51640 10 monclient(hunting): authenticate will time out at 1286.398509s 2023-11-02T08:35:34.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800dd90 0x7fbd5c139830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800dd90 0x7fbd5c139830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:42614/0 (socket says 172.21.15.178:42614) 2023-11-02T08:35:34.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 1 -- 172.21.15.178:0/3761324851 learned_addr learned my addr 172.21.15.178:0/3761324851 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-02T08:35:34.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 10 monclient(hunting): get_auth_request con 0x7fbd4800dd90 auth_method 0 2023-11-02T08:35:34.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:34.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:34.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:34.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:34.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.073+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:34.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.075+0000 7fbd5a59c640 1 --2- 172.21.15.178:0/3761324851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 0x7fbd5c0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5a59c640 10 monclient(hunting): get_auth_request con 0x7fbd48003970 auth_method 0 2023-11-02T08:35:34.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:34.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5a59c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5a59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:34.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2023-11-02T08:35:34.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 1 -- 172.21.15.178:0/3761324851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800aa00 msgr2=0x7fbd5c1392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 1 --2- 172.21.15.178:0/3761324851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800aa00 0x7fbd5c1392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 1 -- 172.21.15.178:0/3761324851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 msgr2=0x7fbd5c0aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 1 --2- 172.21.15.178:0/3761324851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 0x7fbd5c0aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 10 monclient: _finish_hunting 0 2023-11-02T08:35:34.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 1 monclient: found mon.noname-b 2023-11-02T08:35:34.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:34.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:35:34.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.076+0000 7fbd5ad9d640 1 -- 172.21.15.178:0/3761324851 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c1c9e90 con 0x7fbd4800dd90 2023-11-02T08:35:34.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.078+0000 7fbd5ad9d640 1 --2- 172.21.15.178:0/3761324851 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800dd90 0x7fbd5c139830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbd54016040 tx=0x7fbd54009750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:34.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.078+0000 7fbd52ffd640 10 client.?.objecter ms_handle_connect 0x7fbd4800dd90 2023-11-02T08:35:34.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.078+0000 7fbd52ffd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:34.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 1 -- 172.21.15.178:0/3761324851 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54004510 con 0x7fbd4800dd90 2023-11-02T08:35:34.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:34.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:35:34.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 10 monclient: dump: 2023-11-02T08:35:34.104 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:34.104 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.104 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.105 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.105 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:34.105 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.105 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:34.105 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:34.106 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 10 monclient: _finish_auth 0 2023-11-02T08:35:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 10 monclient: _check_auth_tickets 2023-11-02T08:35:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 1 -- 172.21.15.178:0/3761324851 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd54004670 con 0x7fbd4800dd90 2023-11-02T08:35:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 1 -- 172.21.15.178:0/3761324851 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54026460 con 0x7fbd4800dd90 2023-11-02T08:35:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd60e51640 5 monclient: authenticate success, global_id 4117 2023-11-02T08:35:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd60e51640 10 monclient: _renew_subs 2023-11-02T08:35:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd60e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd60e51640 1 -- 172.21.15.178:0/3761324851 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c0af050 con 0x7fbd4800dd90 2023-11-02T08:35:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd5959a640 4 set_mon_vals no callback set 2023-11-02T08:35:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd52ffd640 10 monclient: dump: 2023-11-02T08:35:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.111 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:34.111 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:34.111 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:34.111 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd60e51640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:34.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd60e51640 10 monclient: _renew_subs 2023-11-02T08:35:34.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd60e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:34.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd60e51640 1 -- 172.21.15.178:0/3761324851 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c1cacb0 con 0x7fbd4800dd90 2023-11-02T08:35:34.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.088+0000 7fbd60e51640 1 librados: init done 2023-11-02T08:35:34.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.089+0000 7fbd60e51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:34.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.089+0000 7fbd60e51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:34.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.089+0000 7fbd60e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:34.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.089+0000 7fbd60e51640 1 -- 172.21.15.178:0/3761324851 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c0af240 con 0x7fbd4800dd90 2023-11-02T08:35:34.214 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.211+0000 7fbd52ffd640 1 -- 172.21.15.178:0/3761324851 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbd540268e0 con 0x7fbd4800dd90 2023-11-02T08:35:34.214 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.211+0000 7fbd52ffd640 1 -- 172.21.15.178:0/3761324851 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbd540148a0 con 0x7fbd4800dd90 2023-11-02T08:35:34.215 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.211+0000 7fbd52ffd640 10 client.4117.objecter ms_dispatch 0x7fbd5c000b90 osd_map(2..2 src has 1..2) v4 2023-11-02T08:35:34.215 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.211+0000 7fbd52ffd640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2023-11-02T08:35:34.215 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.211+0000 7fbd52ffd640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2023-11-02T08:35:34.215 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.212+0000 7fbd52ffd640 20 client.4117.objecter dump_active .. 0 homeless 2023-11-02T08:35:34.216 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.212+0000 7fbd52ffd640 1 -- 172.21.15.178:0/3761324851 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fbd54021070 con 0x7fbd4800dd90 2023-11-02T08:35:34.216 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.212+0000 7fbd52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:34.216 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.212+0000 7fbd52ffd640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:34.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.283+0000 7fbd60e51640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fed323c8-4fdc-46c5-bf48-ba99f939a935", "id": 0}] 2023-11-02T08:35:34.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.283+0000 7fbd60e51640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fed323c8-4fdc-46c5-bf48-ba99f939a935", "id": 0}] 2023-11-02T08:35:34.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.283+0000 7fbd60e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:34.287 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.283+0000 7fbd60e51640 1 -- 172.21.15.178:0/3761324851 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fed323c8-4fdc-46c5-bf48-ba99f939a935", "id": 0} v 0) v1 -- 0x7fbd5c0abf80 con 0x7fbd4800dd90 2023-11-02T08:35:34.389 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:34.386+0000 7fb1fc810040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-02T08:35:34.437 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:34.434+0000 7f0836c1f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-02T08:35:34.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.453+0000 7fbd52ffd640 1 -- 172.21.15.178:0/3761324851 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fed323c8-4fdc-46c5-bf48-ba99f939a935", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbd54009ca0 con 0x7fbd4800dd90 2023-11-02T08:35:34.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.453+0000 7fbd52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fed323c8-4fdc-46c5-bf48-ba99f939a935", "id": 0}] 2023-11-02T08:35:34.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.453+0000 7fbd52ffd640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:34.456 INFO:teuthology.orchestra.run.smithi178.stdout:0 2023-11-02T08:35:34.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 10 librados: watch_flush enter 2023-11-02T08:35:34.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 10 librados: watch_flush exit 2023-11-02T08:35:34.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 20 client.4117.objecter shutdown clearing up homeless session... 2023-11-02T08:35:34.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 10 client.4117.objecter successfully canceled tick 2023-11-02T08:35:34.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 10 monclient: shutdown 2023-11-02T08:35:34.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:34.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 -- 172.21.15.178:0/3761324851 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800dd90 msgr2=0x7fbd5c139830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 --2- 172.21.15.178:0/3761324851 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800dd90 0x7fbd5c139830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbd54016040 tx=0x7fbd54009750 comp rx=0 tx=0).stop 2023-11-02T08:35:34.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 -- 172.21.15.178:0/3761324851 shutdown_connections 2023-11-02T08:35:34.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 --2- 172.21.15.178:0/3761324851 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd4800dd90 0x7fbd5c139830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 --2- 172.21.15.178:0/3761324851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd4800aa00 0x7fbd5c1392f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 --2- 172.21.15.178:0/3761324851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd48003970 0x7fbd5c0aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 -- 172.21.15.178:0/3761324851 >> 172.21.15.178:0/3761324851 conn(0x7fbd5c065f90 msgr2=0x7fbd5c13c870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:34.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 -- 172.21.15.178:0/3761324851 shutdown_connections 2023-11-02T08:35:34.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 -- 172.21.15.178:0/3761324851 wait complete. 2023-11-02T08:35:34.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.454+0000 7fbd60e51640 1 librados: shutdown 2023-11-02T08:35:34.467 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 606297f6-206d-441d-99cd-3f50f334b5ac 1 2023-11-02T08:35:34.532 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:34.529+0000 7fb1fc810040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-02T08:35:34.573 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:34.569+0000 7f0836c1f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-02T08:35:34.606 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.604+0000 7f247a077640 10 monclient: get_monmap_and_config 2023-11-02T08:35:34.607 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.604+0000 7f247a077640 10 monclient: build_initial_monmap 2023-11-02T08:35:34.607 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.604+0000 7f247a077640 10 monclient: monmap: 2023-11-02T08:35:34.607 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:34.608 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.608 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:35:34.604383+0000 2023-11-02T08:35:34.608 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:35:34.604383+0000 2023-11-02T08:35:34.608 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:34.609 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.609 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:34.609 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:34.609 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:34.610 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:34.610 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:34.610 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.613 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 1 Processor -- start 2023-11-02T08:35:34.613 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 1 -- start start 2023-11-02T08:35:34.613 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 10 monclient: init 2023-11-02T08:35:34.613 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:34.614 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-11-02T08:35:34.614 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24740b4570 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:34.614 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f24740b0ed0 0x7f2474144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.614 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 10 monclient(hunting): picked mon.noname-a con 0x7f24740b0ed0 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:34.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f247413e520 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:34.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24740a9780 con 0x7f247413e520 2023-11-02T08:35:34.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24740aa1d0 con 0x7f24740b4570 2023-11-02T08:35:34.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24740b4950 con 0x7f24740b0ed0 2023-11-02T08:35:34.616 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:34.616 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.605+0000 7f247a077640 10 monclient(hunting): authenticate will time out at 1286.931724s 2023-11-02T08:35:34.616 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.606+0000 7f2472ffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f24740b0ed0 0x7f2474144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.616 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.606+0000 7f24737fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f24740b4570 0x7f24740a9070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:39970/0 (socket says 172.21.15.178:39970) 2023-11-02T08:35:34.617 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.606+0000 7f24737fe640 1 -- 172.21.15.178:0/1320270741 learned_addr learned my addr 172.21.15.178:0/1320270741 (peer_addr_for_me v1:172.21.15.178:0/0) 2023-11-02T08:35:34.617 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.606+0000 7f2472ffd640 10 monclient(hunting): get_auth_request con 0x7f24740b0ed0 auth_method 0 2023-11-02T08:35:34.617 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.606+0000 7f2472ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:34.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.606+0000 7f2472ffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.606+0000 7f2472ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:34.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.607+0000 7f2472ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:34.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.607+0000 7f2472ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:34.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.607+0000 7f2472ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:34.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f24727fc640 1 -- 172.21.15.178:0/1320270741 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288579686 0 0) 0x7f24740aa1d0 con 0x7f24740b4570 2023-11-02T08:35:34.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f24727fc640 10 monclient(hunting): _negotiate 2023-11-02T08:35:34.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f24727fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f24727fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:34.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f24727fc640 10 monclient(hunting): my global_id is 4120 2023-11-02T08:35:34.620 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f24727fc640 1 -- 172.21.15.178:0/1320270741 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f245c003520 con 0x7f24740b4570 2023-11-02T08:35:34.620 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f2472ffd640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2023-11-02T08:35:34.621 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f2472ffd640 1 -- 172.21.15.178:0/1320270741 >> v1:172.21.15.178:6789/0 conn(0x7f24740b4570 legacy=0x7f24740a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.621 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f2472ffd640 1 -- 172.21.15.178:0/1320270741 >> v1:172.21.15.151:6789/0 conn(0x7f247413e520 legacy=0x7f247413e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:34.621 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f2472ffd640 10 monclient: _finish_hunting 0 2023-11-02T08:35:34.621 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f2472ffd640 1 monclient: found mon.noname-a 2023-11-02T08:35:34.621 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f2472ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:34.622 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f2472ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:34.622 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.609+0000 7f2472ffd640 1 -- 172.21.15.178:0/1320270741 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24740aa4b0 con 0x7f24740b0ed0 2023-11-02T08:35:34.629 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.613+0000 7f2472ffd640 1 --2- 172.21.15.178:0/1320270741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 0x7f2474144d40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f24680097c0 tx=0x7f246802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fe4219e155e70 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:34.629 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 1 -- 172.21.15.178:0/1320270741 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24680316f0 con 0x7f24740b0ed0 2023-11-02T08:35:34.629 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:34.630 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:34.630 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 10 monclient: dump: 2023-11-02T08:35:34.630 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:34.631 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.631 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.631 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.632 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:34.632 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.632 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:34.632 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:34.632 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:34.633 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.633 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 10 monclient: _finish_auth 0 2023-11-02T08:35:34.633 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 10 monclient: _check_auth_tickets 2023-11-02T08:35:34.633 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:34.633 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 1 -- 172.21.15.178:0/1320270741 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2468031850 con 0x7f24740b0ed0 2023-11-02T08:35:34.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:34.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 1 -- 172.21.15.178:0/1320270741 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468031be0 con 0x7f24740b0ed0 2023-11-02T08:35:34.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 5 monclient: authenticate success, global_id 4113 2023-11-02T08:35:34.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:34.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:34.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 4 set_mon_vals no callback set 2023-11-02T08:35:34.636 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:34.636 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f24727fc640 10 monclient: dump: 2023-11-02T08:35:34.636 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:34.636 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.636 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.637 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.637 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:34.637 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.637 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:34.637 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:34.638 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:34.638 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.638 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 10 monclient: shutdown 2023-11-02T08:35:34.638 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:34.639 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 1 -- 172.21.15.178:0/1320270741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 msgr2=0x7f2474144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.639 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 1 --2- 172.21.15.178:0/1320270741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 0x7f2474144d40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f24680097c0 tx=0x7f246802b790 comp rx=0 tx=0).stop 2023-11-02T08:35:34.639 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 1 -- 172.21.15.178:0/1320270741 shutdown_connections 2023-11-02T08:35:34.640 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 1 --2- 172.21.15.178:0/1320270741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 0x7f2474144d40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.640 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 1 -- 172.21.15.178:0/1320270741 >> 172.21.15.178:0/1320270741 conn(0x7f2474065d50 msgr2=0x7f2474066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:34.640 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 1 -- 172.21.15.178:0/1320270741 shutdown_connections 2023-11-02T08:35:34.640 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.618+0000 7f247a077640 1 -- 172.21.15.178:0/1320270741 wait complete. 2023-11-02T08:35:34.641 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.619+0000 7f247a077640 10 monclient: build_initial_monmap 2023-11-02T08:35:34.641 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.619+0000 7f247a077640 10 monclient: monmap: 2023-11-02T08:35:34.641 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:34.641 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:34.641 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2023-11-02T08:35:34.642 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2023-11-02T08:35:34.642 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:34.642 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.642 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:34.642 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:34.643 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:34.643 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.643 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.619+0000 7f247a077640 1 librados: starting msgr at 2023-11-02T08:35:34.643 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.619+0000 7f247a077640 1 librados: starting objecter 2023-11-02T08:35:34.644 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.619+0000 7f247a077640 1 Processor -- start 2023-11-02T08:35:34.644 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 1 -- start start 2023-11-02T08:35:34.644 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 1 librados: setting wanted keys 2023-11-02T08:35:34.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 1 librados: calling monclient init 2023-11-02T08:35:34.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient: init 2023-11-02T08:35:34.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:34.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:35:34.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 0x7f2474144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient(hunting): picked mon.noname-a con 0x7f24740b0ed0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:34.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24740b4570 0x7f24741d9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient(hunting): picked mon.noname-c con 0x7f24740b4570 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:34.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f247413e520 0x7f24741d6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:34.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient(hunting): picked mon.noname-b con 0x7f247413e520 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:34.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2474141e30 con 0x7f24740b0ed0 2023-11-02T08:35:34.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741d9e00 con 0x7f247413e520 2023-11-02T08:35:34.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:34.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24741d6770 con 0x7f24740b4570 2023-11-02T08:35:34.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:34.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.620+0000 7f247a077640 10 monclient(hunting): authenticate will time out at 1286.946821s 2023-11-02T08:35:34.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f24737fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 0x7f2474144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f2473fff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f247413e520 0x7f24741d6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f2472ffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24740b4570 0x7f24741d9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:34.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f24737fe640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 0x7f2474144900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.178:35778/0 (socket says 172.21.15.178:35778) 2023-11-02T08:35:34.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f24737fe640 1 -- 172.21.15.178:0/1750437718 learned_addr learned my addr 172.21.15.178:0/1750437718 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-02T08:35:34.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f24737fe640 10 monclient(hunting): get_auth_request con 0x7f24740b0ed0 auth_method 0 2023-11-02T08:35:34.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f24737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:34.651 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f24737fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.651 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f24737fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:34.651 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f2473fff640 10 monclient(hunting): get_auth_request con 0x7f247413e520 auth_method 0 2023-11-02T08:35:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f2473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f2473fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.621+0000 7f2473fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2472ffd640 10 monclient(hunting): get_auth_request con 0x7f24740b4570 auth_method 0 2023-11-02T08:35:34.652 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2472ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:34.653 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2472ffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:34.653 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2472ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:34.653 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f24737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:34.653 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f24737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:34.653 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f24737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:34.654 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2473fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:34.654 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2473fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:34.654 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2473fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:34.654 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2472ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:34.654 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2472ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.622+0000 7f2472ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 10 monclient(hunting): handle_auth_done global_id 4116 payload 824 2023-11-02T08:35:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 1 -- 172.21.15.178:0/1750437718 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24740b4570 msgr2=0x7f24741d9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 1 --2- 172.21.15.178:0/1750437718 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24740b4570 0x7f24741d9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.655 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 1 -- 172.21.15.178:0/1750437718 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f247413e520 msgr2=0x7f24741d6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.656 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 1 --2- 172.21.15.178:0/1750437718 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f247413e520 0x7f24741d6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.656 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 10 monclient: _finish_hunting 0 2023-11-02T08:35:34.656 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 1 monclient: found mon.noname-a 2023-11-02T08:35:34.657 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:34.657 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:34.657 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.624+0000 7f24737fe640 1 -- 172.21.15.178:0/1750437718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24680093a0 con 0x7f24740b0ed0 2023-11-02T08:35:34.657 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.627+0000 7f24737fe640 1 --2- 172.21.15.178:0/1750437718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 0x7f2474144900 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f246400c500 tx=0x7f246400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:34.657 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.627+0000 7f245bfff640 10 client.?.objecter ms_handle_connect 0x7f24740b0ed0 2023-11-02T08:35:34.658 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.627+0000 7f245bfff640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:34.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 1 -- 172.21.15.178:0/1750437718 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464005ce0 con 0x7f24740b0ed0 2023-11-02T08:35:34.660 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:34.660 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:34.660 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 10 monclient: dump: 2023-11-02T08:35:34.660 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:34.660 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.661 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.661 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.661 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:34.661 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.662 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:34.662 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:34.662 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:34.662 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 10 monclient: _finish_auth 0 2023-11-02T08:35:34.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 10 monclient: _check_auth_tickets 2023-11-02T08:35:34.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:34.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 1 -- 172.21.15.178:0/1750437718 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464005e40 con 0x7f24740b0ed0 2023-11-02T08:35:34.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:34.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 1 -- 172.21.15.178:0/1750437718 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464007330 con 0x7f24740b0ed0 2023-11-02T08:35:34.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:34.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:34.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f245bfff640 10 monclient: dump: 2023-11-02T08:35:34.664 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:34.665 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:34.665 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.665 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:34.665 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:34.666 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:34.666 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:34.666 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:34.667 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:34.667 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:34.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f247a077640 5 monclient: authenticate success, global_id 4116 2023-11-02T08:35:34.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f247a077640 10 monclient: _renew_subs 2023-11-02T08:35:34.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f247a077640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:34.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f247a077640 1 -- 172.21.15.178:0/1750437718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24740b0920 con 0x7f24740b0ed0 2023-11-02T08:35:34.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f2471ffb640 4 set_mon_vals no callback set 2023-11-02T08:35:34.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f247a077640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:34.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f247a077640 10 monclient: _renew_subs 2023-11-02T08:35:34.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f247a077640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:34.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f247a077640 1 -- 172.21.15.178:0/1750437718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24740b0bd0 con 0x7f24740b0ed0 2023-11-02T08:35:34.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.631+0000 7f247a077640 1 librados: init done 2023-11-02T08:35:34.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.632+0000 7f247a077640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:34.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.632+0000 7f247a077640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:34.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.632+0000 7f247a077640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:34.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.632+0000 7f247a077640 1 -- 172.21.15.178:0/1750437718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24741d6990 con 0x7f24740b0ed0 2023-11-02T08:35:34.672 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:34.643+0000 7fb1fc810040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-02T08:35:34.691 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:34.688+0000 7fb1fc810040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-02T08:35:34.723 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:34.720+0000 7f0836c1f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-02T08:35:34.769 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:34.766+0000 7f0836c1f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-02T08:35:34.773 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.771+0000 7f245bfff640 1 -- 172.21.15.178:0/1750437718 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f246401c440 con 0x7f24740b0ed0 2023-11-02T08:35:34.778 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.776+0000 7f245bfff640 1 -- 172.21.15.178:0/1750437718 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f2464012880 con 0x7f24740b0ed0 2023-11-02T08:35:34.779 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.776+0000 7f245bfff640 10 client.4116.objecter ms_dispatch 0x7f2474000b90 osd_map(3..3 src has 1..3) v4 2023-11-02T08:35:34.779 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.776+0000 7f245bfff640 3 client.4116.objecter handle_osd_map got epochs [3,3] > 0 2023-11-02T08:35:34.779 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.776+0000 7f245bfff640 3 client.4116.objecter handle_osd_map decoding full epoch 3 2023-11-02T08:35:34.779 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.776+0000 7f245bfff640 20 client.4116.objecter dump_active .. 0 homeless 2023-11-02T08:35:34.780 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.776+0000 7f245bfff640 1 -- 172.21.15.178:0/1750437718 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f2464017070 con 0x7f24740b0ed0 2023-11-02T08:35:34.780 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.776+0000 7f245bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:34.780 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.776+0000 7f245bfff640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:34.788 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:34.786+0000 7fb1fc810040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-02T08:35:34.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.847+0000 7f247a077640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "606297f6-206d-441d-99cd-3f50f334b5ac", "id": 1}] 2023-11-02T08:35:34.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.847+0000 7f247a077640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "606297f6-206d-441d-99cd-3f50f334b5ac", "id": 1}] 2023-11-02T08:35:34.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.847+0000 7f247a077640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:34.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.847+0000 7f247a077640 1 -- 172.21.15.178:0/1750437718 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "606297f6-206d-441d-99cd-3f50f334b5ac", "id": 1} v 0) v1 -- 0x7f2474067920 con 0x7f24740b0ed0 2023-11-02T08:35:34.866 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:34.863+0000 7f0836c1f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-02T08:35:34.884 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:34.882+0000 7fb1fc810040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-02T08:35:34.911 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.908+0000 7f245bfff640 1 -- 172.21.15.178:0/1750437718 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "606297f6-206d-441d-99cd-3f50f334b5ac", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2464012b70 con 0x7f24740b0ed0 2023-11-02T08:35:34.911 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.908+0000 7f245bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "606297f6-206d-441d-99cd-3f50f334b5ac", "id": 1}] 2023-11-02T08:35:34.912 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.908+0000 7f245bfff640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:34.912 INFO:teuthology.orchestra.run.smithi178.stdout:1 2023-11-02T08:35:34.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.909+0000 7f247a077640 10 librados: watch_flush enter 2023-11-02T08:35:34.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.909+0000 7f247a077640 10 librados: watch_flush exit 2023-11-02T08:35:34.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.909+0000 7f247a077640 20 client.4116.objecter shutdown clearing up homeless session... 2023-11-02T08:35:34.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.909+0000 7f247a077640 10 client.4116.objecter successfully canceled tick 2023-11-02T08:35:34.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.909+0000 7f247a077640 10 monclient: shutdown 2023-11-02T08:35:34.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.909+0000 7f247a077640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:34.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.909+0000 7f247a077640 1 -- 172.21.15.178:0/1750437718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 msgr2=0x7f2474144900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:34.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.909+0000 7f247a077640 1 --2- 172.21.15.178:0/1750437718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 0x7f2474144900 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f246400c500 tx=0x7f246400c9c0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.909+0000 7f247a077640 1 -- 172.21.15.178:0/1750437718 shutdown_connections 2023-11-02T08:35:34.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.910+0000 7f247a077640 1 --2- 172.21.15.178:0/1750437718 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f247413e520 0x7f24741d6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.910+0000 7f247a077640 1 --2- 172.21.15.178:0/1750437718 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24740b4570 0x7f24741d9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.910+0000 7f247a077640 1 --2- 172.21.15.178:0/1750437718 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24740b0ed0 0x7f2474144900 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:34.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.910+0000 7f247a077640 1 -- 172.21.15.178:0/1750437718 >> 172.21.15.178:0/1750437718 conn(0x7f2474065d50 msgr2=0x7f24740b72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:34.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.910+0000 7f247a077640 1 -- 172.21.15.178:0/1750437718 shutdown_connections 2023-11-02T08:35:34.917 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.910+0000 7f247a077640 1 -- 172.21.15.178:0/1750437718 wait complete. 2023-11-02T08:35:34.917 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:34.910+0000 7f247a077640 1 librados: shutdown 2023-11-02T08:35:34.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 070974a7-a457-460e-a323-d4685aa2950e 2 2023-11-02T08:35:34.982 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:34.978+0000 7f0836c1f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-02T08:35:35.034 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:35.032+0000 7fb1fc810040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-02T08:35:35.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.060+0000 7f5c166e5640 10 monclient: get_monmap_and_config 2023-11-02T08:35:35.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.060+0000 7f5c166e5640 10 monclient: build_initial_monmap 2023-11-02T08:35:35.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.060+0000 7f5c166e5640 10 monclient: monmap: 2023-11-02T08:35:35.063 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:35.063 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.064 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:35:35.060370+0000 2023-11-02T08:35:35.064 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:35:35.060370+0000 2023-11-02T08:35:35.064 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:35.064 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.064 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:35.064 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:35.065 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:35.065 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:35.065 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:35.065 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.069 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.060+0000 7f5c166e5640 1 Processor -- start 2023-11-02T08:35:35.070 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 1 -- start start 2023-11-02T08:35:35.070 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 10 monclient: init 2023-11-02T08:35:35.070 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:35.070 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-11-02T08:35:35.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c100b4570 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:35.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c100b0ed0 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:35.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c10144d40 0x7f5c1013ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 10 monclient(hunting): picked mon.noname-b con 0x7f5c10144d40 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:35.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c100a9780 con 0x7f5c100b0ed0 2023-11-02T08:35:35.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c101451d0 con 0x7f5c100b4570 2023-11-02T08:35:35.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100aa1d0 con 0x7f5c10144d40 2023-11-02T08:35:35.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:35.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.061+0000 7f5c166e5640 10 monclient(hunting): authenticate will time out at 1287.387685s 2023-11-02T08:35:35.073 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.062+0000 7f5c0ffff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f5c100b4570 0x7f5c100a9070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:39986/0 (socket says 172.21.15.178:39986) 2023-11-02T08:35:35.073 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.062+0000 7f5c0ffff640 1 -- 172.21.15.178:0/3108069322 learned_addr learned my addr 172.21.15.178:0/3108069322 (peer_addr_for_me v1:172.21.15.178:0/0) 2023-11-02T08:35:35.073 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.062+0000 7f5c0f7fe640 1 --1- 172.21.15.178:0/3108069322 >> v1:172.21.15.151:6789/0 conn(0x7f5c100b0ed0 0x7f5c1013e520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.178:49954/0 (socket says 172.21.15.178:49954) 2023-11-02T08:35:35.073 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.062+0000 7f5c14c82640 1 --2- 172.21.15.178:0/3108069322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c10144d40 0x7f5c1013ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.073 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.062+0000 7f5c14c82640 10 monclient(hunting): get_auth_request con 0x7f5c10144d40 auth_method 0 2023-11-02T08:35:35.074 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.062+0000 7f5c14c82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.074 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.062+0000 7f5c14c82640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.074 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.062+0000 7f5c14c82640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.074 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.063+0000 7f5c14c82640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.074 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.063+0000 7f5c14c82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.075 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.063+0000 7f5c14c82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.075 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.064+0000 7f5c0effd640 1 -- 172.21.15.178:0/3108069322 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593124720 0 0) 0x7f5c101451d0 con 0x7f5c100b4570 2023-11-02T08:35:35.075 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.064+0000 7f5c0effd640 10 monclient(hunting): _negotiate 2023-11-02T08:35:35.075 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.064+0000 7f5c0effd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.075 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.064+0000 7f5c0effd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.076 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.064+0000 7f5c0effd640 10 monclient(hunting): my global_id is 4126 2023-11-02T08:35:35.076 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.064+0000 7f5c0effd640 1 -- 172.21.15.178:0/3108069322 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf8003520 con 0x7f5c100b4570 2023-11-02T08:35:35.076 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.065+0000 7f5c14c82640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2023-11-02T08:35:35.076 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.065+0000 7f5c14c82640 1 -- 172.21.15.178:0/3108069322 >> v1:172.21.15.178:6789/0 conn(0x7f5c100b4570 legacy=0x7f5c100a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.076 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.065+0000 7f5c14c82640 1 -- 172.21.15.178:0/3108069322 >> v1:172.21.15.151:6789/0 conn(0x7f5c100b0ed0 legacy=0x7f5c1013e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.077 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.065+0000 7f5c14c82640 10 monclient: _finish_hunting 0 2023-11-02T08:35:35.077 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.065+0000 7f5c14c82640 1 monclient: found mon.noname-b 2023-11-02T08:35:35.077 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.065+0000 7f5c14c82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:35.077 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.065+0000 7f5c14c82640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:35:35.077 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.065+0000 7f5c14c82640 1 -- 172.21.15.178:0/3108069322 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c100aa4b0 con 0x7f5c10144d40 2023-11-02T08:35:35.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.067+0000 7f5c14c82640 1 --2- 172.21.15.178:0/3108069322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c10144d40 0x7f5c1013ec30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009df0 tx=0x7f5c0802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aa726b29d1fda45 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:35.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 1 -- 172.21.15.178:0/3108069322 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0802cb90 con 0x7f5c10144d40 2023-11-02T08:35:35.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:35:35.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 10 monclient: dump: 2023-11-02T08:35:35.083 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.083 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.083 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.083 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.083 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.084 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.084 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.084 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.084 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.084 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 10 monclient: _finish_auth 0 2023-11-02T08:35:35.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 10 monclient: _check_auth_tickets 2023-11-02T08:35:35.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:35.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 1 -- 172.21.15.178:0/3108069322 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0802ccf0 con 0x7f5c10144d40 2023-11-02T08:35:35.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c166e5640 5 monclient: authenticate success, global_id 4112 2023-11-02T08:35:35.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c166e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:35:35.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:35.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 1 -- 172.21.15.178:0/3108069322 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08034660 con 0x7f5c10144d40 2023-11-02T08:35:35.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c166e5640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:35.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c166e5640 4 set_mon_vals no callback set 2023-11-02T08:35:35.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:35:35.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.070+0000 7f5c0effd640 10 monclient: dump: 2023-11-02T08:35:35.087 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.088 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.088 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.088 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.088 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.088 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.089 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.089 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.089 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.089 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.071+0000 7f5c166e5640 10 monclient: shutdown 2023-11-02T08:35:35.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.071+0000 7f5c166e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:35.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.071+0000 7f5c166e5640 1 -- 172.21.15.178:0/3108069322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c10144d40 msgr2=0x7f5c1013ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.071+0000 7f5c166e5640 1 --2- 172.21.15.178:0/3108069322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c10144d40 0x7f5c1013ec30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009df0 tx=0x7f5c0802bde0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.071+0000 7f5c166e5640 1 -- 172.21.15.178:0/3108069322 shutdown_connections 2023-11-02T08:35:35.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.071+0000 7f5c166e5640 1 --2- 172.21.15.178:0/3108069322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c10144d40 0x7f5c1013ec30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.071+0000 7f5c166e5640 1 -- 172.21.15.178:0/3108069322 >> 172.21.15.178:0/3108069322 conn(0x7f5c10065d50 msgr2=0x7f5c10066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:35.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.071+0000 7f5c166e5640 1 -- 172.21.15.178:0/3108069322 shutdown_connections 2023-11-02T08:35:35.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.071+0000 7f5c166e5640 1 -- 172.21.15.178:0/3108069322 wait complete. 2023-11-02T08:35:35.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.072+0000 7f5c166e5640 10 monclient: build_initial_monmap 2023-11-02T08:35:35.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.072+0000 7f5c166e5640 10 monclient: monmap: 2023-11-02T08:35:35.092 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:35.092 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:35.092 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2023-11-02T08:35:35.092 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2023-11-02T08:35:35.092 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:35.092 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.093 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:35.093 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:35.093 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:35.093 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.072+0000 7f5c166e5640 1 librados: starting msgr at 2023-11-02T08:35:35.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.072+0000 7f5c166e5640 1 librados: starting objecter 2023-11-02T08:35:35.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.072+0000 7f5c166e5640 1 Processor -- start 2023-11-02T08:35:35.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.072+0000 7f5c166e5640 1 -- start start 2023-11-02T08:35:35.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.072+0000 7f5c166e5640 1 librados: setting wanted keys 2023-11-02T08:35:35.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.072+0000 7f5c166e5640 1 librados: calling monclient init 2023-11-02T08:35:35.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.072+0000 7f5c166e5640 10 monclient: init 2023-11-02T08:35:35.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:35.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-02T08:35:35.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c100b0ed0 0x7f5c10143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient(hunting): picked mon.noname-b con 0x7f5c100b0ed0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:35.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c100b4570 0x7f5c100ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient(hunting): picked mon.noname-c con 0x7f5c100b4570 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:35.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c10144d40 0x7f5c100acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient(hunting): picked mon.noname-a con 0x7f5c10144d40 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:35.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c101440e0 con 0x7f5c10144d40 2023-11-02T08:35:35.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100ad490 con 0x7f5c100b0ed0 2023-11-02T08:35:35.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c10139700 con 0x7f5c100b4570 2023-11-02T08:35:35.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:35.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.073+0000 7f5c166e5640 10 monclient(hunting): authenticate will time out at 1287.399496s 2023-11-02T08:35:35.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0f7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c100b4570 0x7f5c100ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c14c82640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c10144d40 0x7f5c100acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c100b0ed0 0x7f5c10143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c100b0ed0 0x7f5c10143320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:42634/0 (socket says 172.21.15.178:42634) 2023-11-02T08:35:35.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0f7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c100b4570 0x7f5c100ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.178:48642/0 (socket says 172.21.15.178:48642) 2023-11-02T08:35:35.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 1 -- 172.21.15.178:0/4124621466 learned_addr learned my addr 172.21.15.178:0/4124621466 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-02T08:35:35.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 10 monclient(hunting): get_auth_request con 0x7f5c100b0ed0 auth_method 0 2023-11-02T08:35:35.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0f7fe640 10 monclient(hunting): get_auth_request con 0x7f5c100b4570 auth_method 0 2023-11-02T08:35:35.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c14c82640 10 monclient(hunting): get_auth_request con 0x7f5c10144d40 auth_method 0 2023-11-02T08:35:35.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c14c82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c14c82640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c14c82640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.074+0000 7f5c0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.075+0000 7f5c0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.075+0000 7f5c0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.075+0000 7f5c0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.075+0000 7f5c14c82640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.075+0000 7f5c14c82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.075+0000 7f5c14c82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2023-11-02T08:35:35.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 1 -- 172.21.15.178:0/4124621466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c100b4570 msgr2=0x7f5c100ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.104 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:35.077+0000 7fb1fc810040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-02T08:35:35.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 1 --2- 172.21.15.178:0/4124621466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c100b4570 0x7f5c100ac840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 1 -- 172.21.15.178:0/4124621466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c10144d40 msgr2=0x7f5c100acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 1 --2- 172.21.15.178:0/4124621466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c10144d40 0x7f5c100acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 10 monclient: _finish_hunting 0 2023-11-02T08:35:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 1 monclient: found mon.noname-b 2023-11-02T08:35:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:35:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.077+0000 7f5c0ffff640 1 -- 172.21.15.178:0/4124621466 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c08009a70 con 0x7f5c100b0ed0 2023-11-02T08:35:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.079+0000 7f5c0ffff640 1 --2- 172.21.15.178:0/4124621466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c100b0ed0 0x7f5c10143320 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000c500 tx=0x7f5c0000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.079+0000 7f5c0cff9640 10 client.?.objecter ms_handle_connect 0x7f5c100b0ed0 2023-11-02T08:35:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.079+0000 7f5c0cff9640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 1 -- 172.21.15.178:0/4124621466 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00005ce0 con 0x7f5c100b0ed0 2023-11-02T08:35:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:35:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 10 monclient: dump: 2023-11-02T08:35:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.110 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.110 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.110 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.110 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.110 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.111 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.111 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 10 monclient: _finish_auth 0 2023-11-02T08:35:35.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 10 monclient: _check_auth_tickets 2023-11-02T08:35:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 1 -- 172.21.15.178:0/4124621466 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c00005e40 con 0x7f5c100b0ed0 2023-11-02T08:35:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 1 -- 172.21.15.178:0/4124621466 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00007330 con 0x7f5c100b0ed0 2023-11-02T08:35:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c166e5640 5 monclient: authenticate success, global_id 4129 2023-11-02T08:35:35.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c166e5640 10 monclient: _renew_subs 2023-11-02T08:35:35.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c166e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:35.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c166e5640 1 -- 172.21.15.178:0/4124621466 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c10139980 con 0x7f5c100b0ed0 2023-11-02T08:35:35.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0e7fc640 4 set_mon_vals no callback set 2023-11-02T08:35:35.114 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:35.114 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c0cff9640 10 monclient: dump: 2023-11-02T08:35:35.114 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.114 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.115 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.115 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.115 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.115 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.115 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.116 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.116 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.116 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.116 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c166e5640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:35.116 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c166e5640 10 monclient: _renew_subs 2023-11-02T08:35:35.116 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c166e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:35.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c166e5640 1 -- 172.21.15.178:0/4124621466 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c10139e40 con 0x7f5c100b0ed0 2023-11-02T08:35:35.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.083+0000 7f5c166e5640 1 librados: init done 2023-11-02T08:35:35.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.084+0000 7f5c166e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:35.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.084+0000 7f5c166e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:35.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.084+0000 7f5c166e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:35.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.084+0000 7f5c166e5640 1 -- 172.21.15.178:0/4124621466 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c10067920 con 0x7f5c100b0ed0 2023-11-02T08:35:35.128 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:35.125+0000 7f0836c1f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-02T08:35:35.170 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:35.167+0000 7f0836c1f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-02T08:35:35.189 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.187+0000 7f5c0cff9640 1 -- 172.21.15.178:0/4124621466 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5c0001c440 con 0x7f5c100b0ed0 2023-11-02T08:35:35.190 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.187+0000 7f5c0cff9640 1 -- 172.21.15.178:0/4124621466 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f5c0001b7d0 con 0x7f5c100b0ed0 2023-11-02T08:35:35.190 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.187+0000 7f5c0cff9640 10 client.4129.objecter ms_dispatch 0x7f5c10000b90 osd_map(4..4 src has 1..4) v4 2023-11-02T08:35:35.190 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.187+0000 7f5c0cff9640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2023-11-02T08:35:35.190 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.187+0000 7f5c0cff9640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2023-11-02T08:35:35.191 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.187+0000 7f5c0cff9640 20 client.4129.objecter dump_active .. 0 homeless 2023-11-02T08:35:35.191 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.187+0000 7f5c0cff9640 1 -- 172.21.15.178:0/4124621466 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f5c00017070 con 0x7f5c100b0ed0 2023-11-02T08:35:35.191 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.187+0000 7f5c0cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:35.191 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.187+0000 7f5c0cff9640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:35.225 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:35.223+0000 7fb1fc810040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-02T08:35:35.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.260+0000 7f5c166e5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "070974a7-a457-460e-a323-d4685aa2950e", "id": 2}] 2023-11-02T08:35:35.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.260+0000 7f5c166e5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "070974a7-a457-460e-a323-d4685aa2950e", "id": 2}] 2023-11-02T08:35:35.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.260+0000 7f5c166e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:35.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.260+0000 7f5c166e5640 1 -- 172.21.15.178:0/4124621466 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd new", "uuid": "070974a7-a457-460e-a323-d4685aa2950e", "id": 2} v 0) v1 -- 0x7f5c100a9070 con 0x7f5c100b0ed0 2023-11-02T08:35:35.336 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:35.333+0000 7f0836c1f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-02T08:35:35.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c0cff9640 1 -- 172.21.15.178:0/4124621466 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "070974a7-a457-460e-a323-d4685aa2950e", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5c0001b5a0 con 0x7f5c100b0ed0 2023-11-02T08:35:35.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c0cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "070974a7-a457-460e-a323-d4685aa2950e", "id": 2}] 2023-11-02T08:35:35.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c0cff9640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:35.337 INFO:teuthology.orchestra.run.smithi178.stdout:2 2023-11-02T08:35:35.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c166e5640 10 librados: watch_flush enter 2023-11-02T08:35:35.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c166e5640 10 librados: watch_flush exit 2023-11-02T08:35:35.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c166e5640 20 client.4129.objecter shutdown clearing up homeless session... 2023-11-02T08:35:35.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c166e5640 10 client.4129.objecter successfully canceled tick 2023-11-02T08:35:35.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c166e5640 10 monclient: shutdown 2023-11-02T08:35:35.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c166e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:35.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c166e5640 1 -- 172.21.15.178:0/4124621466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c100b0ed0 msgr2=0x7f5c10143320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.334+0000 7f5c166e5640 1 --2- 172.21.15.178:0/4124621466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c100b0ed0 0x7f5c10143320 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000c500 tx=0x7f5c0000c9c0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.335+0000 7f5c166e5640 1 -- 172.21.15.178:0/4124621466 shutdown_connections 2023-11-02T08:35:35.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.335+0000 7f5c166e5640 1 --2- 172.21.15.178:0/4124621466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5c10144d40 0x7f5c100acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.335+0000 7f5c166e5640 1 --2- 172.21.15.178:0/4124621466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5c100b4570 0x7f5c100ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.335+0000 7f5c166e5640 1 --2- 172.21.15.178:0/4124621466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c100b0ed0 0x7f5c10143320 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.335+0000 7f5c166e5640 1 -- 172.21.15.178:0/4124621466 >> 172.21.15.178:0/4124621466 conn(0x7f5c10065d50 msgr2=0x7f5c100b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:35.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.335+0000 7f5c166e5640 1 -- 172.21.15.178:0/4124621466 shutdown_connections 2023-11-02T08:35:35.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.335+0000 7f5c166e5640 1 -- 172.21.15.178:0/4124621466 wait complete. 2023-11-02T08:35:35.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.335+0000 7f5c166e5640 1 librados: shutdown 2023-11-02T08:35:35.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6 3 2023-11-02T08:35:35.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.443+0000 7f0122fab640 10 monclient: get_monmap_and_config 2023-11-02T08:35:35.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.443+0000 7f0122fab640 10 monclient: build_initial_monmap 2023-11-02T08:35:35.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.443+0000 7f0122fab640 10 monclient: monmap: 2023-11-02T08:35:35.446 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:35.447 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.447 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:35:35.443894+0000 2023-11-02T08:35:35.447 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:35:35.443894+0000 2023-11-02T08:35:35.447 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:35.448 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.448 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:35.448 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:35.448 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:35.448 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:35.448 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:35.449 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.451 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.444+0000 7f0122fab640 1 Processor -- start 2023-11-02T08:35:35.451 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.444+0000 7f0122fab640 1 -- start start 2023-11-02T08:35:35.451 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.444+0000 7f0122fab640 10 monclient: init 2023-11-02T08:35:35.451 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:35.452 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-11-02T08:35:35.452 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f011c0b4570 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:35.452 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f011c0b0ed0 0x7f011c144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.453 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 10 monclient(hunting): picked mon.noname-a con 0x7f011c0b0ed0 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:35.453 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f011c13e520 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:35.453 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011c0a9780 con 0x7f011c0b4570 2023-11-02T08:35:35.453 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011c0aa1d0 con 0x7f011c13e520 2023-11-02T08:35:35.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011c0b4950 con 0x7f011c0b0ed0 2023-11-02T08:35:35.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:35.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0122fab640 10 monclient(hunting): authenticate will time out at 1287.771254s 2023-11-02T08:35:35.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0121548640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f011c13e520 0x7f011c13e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:39994/0 (socket says 172.21.15.178:39994) 2023-11-02T08:35:35.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f0121548640 1 -- 172.21.15.178:0/787938077 learned_addr learned my addr 172.21.15.178:0/787938077 (peer_addr_for_me v1:172.21.15.178:0/0) 2023-11-02T08:35:35.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.445+0000 7f011bfff640 1 --2- 172.21.15.178:0/787938077 >> v2:172.21.15.151:3300/0 conn(0x7f011c0b0ed0 0x7f011c144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.446+0000 7f011bfff640 10 monclient(hunting): get_auth_request con 0x7f011c0b0ed0 auth_method 0 2023-11-02T08:35:35.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.446+0000 7f011bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.446+0000 7f011bfff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.446+0000 7f011bfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.447+0000 7f011bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.447+0000 7f011bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.447+0000 7f011bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.448+0000 7f011b7fe640 1 -- 172.21.15.178:0/787938077 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637160159 0 0) 0x7f011c0aa1d0 con 0x7f011c13e520 2023-11-02T08:35:35.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.448+0000 7f011b7fe640 10 monclient(hunting): _negotiate 2023-11-02T08:35:35.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.448+0000 7f011b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.448+0000 7f011b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.448+0000 7f011b7fe640 10 monclient(hunting): my global_id is 4132 2023-11-02T08:35:35.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.448+0000 7f011b7fe640 1 -- 172.21.15.178:0/787938077 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f010c003520 con 0x7f011c13e520 2023-11-02T08:35:35.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.449+0000 7f011bfff640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2023-11-02T08:35:35.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.449+0000 7f011bfff640 1 -- 172.21.15.178:0/787938077 >> v1:172.21.15.178:6789/0 conn(0x7f011c13e520 legacy=0x7f011c13e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.449+0000 7f011bfff640 1 -- 172.21.15.178:0/787938077 >> v1:172.21.15.151:6789/0 conn(0x7f011c0b4570 legacy=0x7f011c0a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:35.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.449+0000 7f011bfff640 10 monclient: _finish_hunting 0 2023-11-02T08:35:35.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.449+0000 7f011bfff640 1 monclient: found mon.noname-a 2023-11-02T08:35:35.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.449+0000 7f011bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:35.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.449+0000 7f011bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:35.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.449+0000 7f011bfff640 1 -- 172.21.15.178:0/787938077 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c0aa4b0 con 0x7f011c0b0ed0 2023-11-02T08:35:35.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.452+0000 7f011bfff640 1 --2- 172.21.15.178:0/787938077 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c0b0ed0 0x7f011c144d40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f01080097c0 tx=0x7f010802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75cd9d82453dd6c server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:35.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.454+0000 7f011b7fe640 1 -- 172.21.15.178:0/787938077 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01080316f0 con 0x7f011c0b0ed0 2023-11-02T08:35:35.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.455+0000 7f011b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.455+0000 7f011b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:35.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.455+0000 7f011b7fe640 10 monclient: dump: 2023-11-02T08:35:35.462 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.462 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.463 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.463 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.463 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.464 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.464 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.464 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.464 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.464 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.455+0000 7f011b7fe640 10 monclient: _finish_auth 0 2023-11-02T08:35:35.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.455+0000 7f011b7fe640 10 monclient: _check_auth_tickets 2023-11-02T08:35:35.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.455+0000 7f011b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:35.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.455+0000 7f0122fab640 5 monclient: authenticate success, global_id 4125 2023-11-02T08:35:35.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.455+0000 7f0122fab640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:35:35.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.456+0000 7f011b7fe640 1 -- 172.21.15.178:0/787938077 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108031850 con 0x7f011c0b0ed0 2023-11-02T08:35:35.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.456+0000 7f011b7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:35.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.456+0000 7f011b7fe640 1 -- 172.21.15.178:0/787938077 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108031bc0 con 0x7f011c0b0ed0 2023-11-02T08:35:35.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.456+0000 7f0122fab640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:35.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.456+0000 7f0122fab640 4 set_mon_vals no callback set 2023-11-02T08:35:35.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.456+0000 7f011b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.456+0000 7f011b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:35.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.456+0000 7f011b7fe640 10 monclient: dump: 2023-11-02T08:35:35.468 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.468 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.468 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.468 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.468 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.469 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.469 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.469 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.469 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.469 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.469 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 10 monclient: shutdown 2023-11-02T08:35:35.470 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:35.470 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 1 -- 172.21.15.178:0/787938077 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c0b0ed0 msgr2=0x7f011c144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.470 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 1 --2- 172.21.15.178:0/787938077 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c0b0ed0 0x7f011c144d40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f01080097c0 tx=0x7f010802b790 comp rx=0 tx=0).stop 2023-11-02T08:35:35.470 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 1 -- 172.21.15.178:0/787938077 shutdown_connections 2023-11-02T08:35:35.470 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 1 --2- 172.21.15.178:0/787938077 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c0b0ed0 0x7f011c144d40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.471 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 1 -- 172.21.15.178:0/787938077 >> 172.21.15.178:0/787938077 conn(0x7f011c065d50 msgr2=0x7f011c066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:35.471 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 1 -- 172.21.15.178:0/787938077 shutdown_connections 2023-11-02T08:35:35.471 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 1 -- 172.21.15.178:0/787938077 wait complete. 2023-11-02T08:35:35.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 10 monclient: build_initial_monmap 2023-11-02T08:35:35.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.457+0000 7f0122fab640 10 monclient: monmap: 2023-11-02T08:35:35.476 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:35.476 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:35.476 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2023-11-02T08:35:35.476 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2023-11-02T08:35:35.476 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:35.477 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.477 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:35.477 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:35.477 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:35.477 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.478 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 1 librados: starting msgr at 2023-11-02T08:35:35.478 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 1 librados: starting objecter 2023-11-02T08:35:35.478 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 1 Processor -- start 2023-11-02T08:35:35.478 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 1 -- start start 2023-11-02T08:35:35.478 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 1 librados: setting wanted keys 2023-11-02T08:35:35.479 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 1 librados: calling monclient init 2023-11-02T08:35:35.479 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 10 monclient: init 2023-11-02T08:35:35.479 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:35.479 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-02T08:35:35.479 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.458+0000 7f0122fab640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f011c0b0ed0 0x7f011c142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 10 monclient(hunting): picked mon.noname-b con 0x7f011c0b0ed0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:35.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c0b4570 0x7f011c139540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 10 monclient(hunting): picked mon.noname-c con 0x7f011c0b4570 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:35.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c13e520 0x7f011c139a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 10 monclient(hunting): picked mon.noname-a con 0x7f011c13e520 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:35.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c139fc0 con 0x7f011c13e520 2023-11-02T08:35:35.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c0ac090 con 0x7f011c0b0ed0 2023-11-02T08:35:35.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c0acd60 con 0x7f011c0b4570 2023-11-02T08:35:35.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:35.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0122fab640 10 monclient(hunting): authenticate will time out at 1287.785110s 2023-11-02T08:35:35.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0121548640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c13e520 0x7f011c139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f0120d47640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f011c0b0ed0 0x7f011c142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.459+0000 7f011bfff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c0b4570 0x7f011c139540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0121548640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c13e520 0x7f011c139a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.178:35818/0 (socket says 172.21.15.178:35818) 2023-11-02T08:35:35.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0121548640 1 -- 172.21.15.178:0/1663485061 learned_addr learned my addr 172.21.15.178:0/1663485061 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-02T08:35:35.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0121548640 10 monclient(hunting): get_auth_request con 0x7f011c13e520 auth_method 0 2023-11-02T08:35:35.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0121548640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0121548640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0121548640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f011bfff640 10 monclient(hunting): get_auth_request con 0x7f011c0b4570 auth_method 0 2023-11-02T08:35:35.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f011bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f011bfff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f011bfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0120d47640 10 monclient(hunting): get_auth_request con 0x7f011c0b0ed0 auth_method 0 2023-11-02T08:35:35.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0120d47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0120d47640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0120d47640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0121548640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0121548640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0121548640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0120d47640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0120d47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f0120d47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f011bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f011bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.460+0000 7f011bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 10 monclient(hunting): handle_auth_done global_id 4128 payload 824 2023-11-02T08:35:35.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 1 -- 172.21.15.178:0/1663485061 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c0b4570 msgr2=0x7f011c139540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 1 --2- 172.21.15.178:0/1663485061 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c0b4570 0x7f011c139540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 1 -- 172.21.15.178:0/1663485061 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f011c0b0ed0 msgr2=0x7f011c142cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 1 --2- 172.21.15.178:0/1663485061 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f011c0b0ed0 0x7f011c142cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 10 monclient: _finish_hunting 0 2023-11-02T08:35:35.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 1 monclient: found mon.noname-a 2023-11-02T08:35:35.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:35.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:35.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.463+0000 7f0121548640 1 -- 172.21.15.178:0/1663485061 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080093a0 con 0x7f011c13e520 2023-11-02T08:35:35.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.465+0000 7f0121548640 1 --2- 172.21.15.178:0/1663485061 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c13e520 0x7f011c139a80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f011400cad0 tx=0x7f011400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:35.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.465+0000 7f01197fa640 10 client.?.objecter ms_handle_connect 0x7f011c13e520 2023-11-02T08:35:35.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.465+0000 7f01197fa640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:35.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 1 -- 172.21.15.178:0/1663485061 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011400ac30 con 0x7f011c13e520 2023-11-02T08:35:35.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:35.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 10 monclient: dump: 2023-11-02T08:35:35.491 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.491 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.491 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.491 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.491 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.492 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.492 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.492 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.492 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.492 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 10 monclient: _finish_auth 0 2023-11-02T08:35:35.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 10 monclient: _check_auth_tickets 2023-11-02T08:35:35.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:35.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 1 -- 172.21.15.178:0/1663485061 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011400ad90 con 0x7f011c13e520 2023-11-02T08:35:35.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f0122fab640 5 monclient: authenticate success, global_id 4128 2023-11-02T08:35:35.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f0122fab640 10 monclient: _renew_subs 2023-11-02T08:35:35.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f0122fab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:35.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f0122fab640 1 -- 172.21.15.178:0/1663485061 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011c143940 con 0x7f011c13e520 2023-11-02T08:35:35.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:35.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 1 -- 172.21.15.178:0/1663485061 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0114012620 con 0x7f011c13e520 2023-11-02T08:35:35.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:35.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f01197fa640 10 monclient: dump: 2023-11-02T08:35:35.495 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.495 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.495 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.496 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.496 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.496 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.496 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.496 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.497 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.497 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f011affd640 4 set_mon_vals no callback set 2023-11-02T08:35:35.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f0122fab640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:35.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f0122fab640 10 monclient: _renew_subs 2023-11-02T08:35:35.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f0122fab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:35.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f0122fab640 1 -- 172.21.15.178:0/1663485061 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011c0ad570 con 0x7f011c13e520 2023-11-02T08:35:35.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.469+0000 7f0122fab640 1 librados: init done 2023-11-02T08:35:35.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.470+0000 7f0122fab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:35.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.470+0000 7f0122fab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:35.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.470+0000 7f0122fab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:35.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.470+0000 7f0122fab640 1 -- 172.21.15.178:0/1663485061 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011c0acf80 con 0x7f011c13e520 2023-11-02T08:35:35.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.471+0000 7f01197fa640 1 -- 172.21.15.178:0/1663485061 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0114012a80 con 0x7f011c13e520 2023-11-02T08:35:35.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.471+0000 7f01197fa640 1 -- 172.21.15.178:0/1663485061 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f0114007500 con 0x7f011c13e520 2023-11-02T08:35:35.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.471+0000 7f01197fa640 10 client.4128.objecter ms_dispatch 0x7f011c000b90 osd_map(5..5 src has 1..5) v4 2023-11-02T08:35:35.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.471+0000 7f01197fa640 3 client.4128.objecter handle_osd_map got epochs [5,5] > 0 2023-11-02T08:35:35.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.471+0000 7f01197fa640 3 client.4128.objecter handle_osd_map decoding full epoch 5 2023-11-02T08:35:35.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.471+0000 7f01197fa640 20 client.4128.objecter dump_active .. 0 homeless 2023-11-02T08:35:35.575 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.573+0000 7f01197fa640 1 -- 172.21.15.178:0/1663485061 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f0114017070 con 0x7f011c13e520 2023-11-02T08:35:35.575 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.573+0000 7f01197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:35.576 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.573+0000 7f01197fa640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:35.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.644+0000 7f0122fab640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6", "id": 3}] 2023-11-02T08:35:35.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.644+0000 7f0122fab640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6", "id": 3}] 2023-11-02T08:35:35.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.644+0000 7f0122fab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:35.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.644+0000 7f0122fab640 1 -- 172.21.15.178:0/1663485061 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6", "id": 3} v 0) v1 -- 0x7f011c067920 con 0x7f011c13e520 2023-11-02T08:35:35.693 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.691+0000 7f01197fa640 1 -- 172.21.15.178:0/1663485061 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f011401c5f0 con 0x7f011c13e520 2023-11-02T08:35:35.694 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.691+0000 7f01197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6", "id": 3}] 2023-11-02T08:35:35.694 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.691+0000 7f01197fa640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:35.695 INFO:teuthology.orchestra.run.smithi178.stdout:3 2023-11-02T08:35:35.696 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 10 librados: watch_flush enter 2023-11-02T08:35:35.696 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 10 librados: watch_flush exit 2023-11-02T08:35:35.696 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 20 client.4128.objecter shutdown clearing up homeless session... 2023-11-02T08:35:35.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 10 client.4128.objecter successfully canceled tick 2023-11-02T08:35:35.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 10 monclient: shutdown 2023-11-02T08:35:35.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:35.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 -- 172.21.15.178:0/1663485061 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c13e520 msgr2=0x7f011c139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 --2- 172.21.15.178:0/1663485061 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c13e520 0x7f011c139a80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f011400cad0 tx=0x7f011400cf90 comp rx=0 tx=0).stop 2023-11-02T08:35:35.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 -- 172.21.15.178:0/1663485061 shutdown_connections 2023-11-02T08:35:35.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 --2- 172.21.15.178:0/1663485061 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c13e520 0x7f011c139a80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 --2- 172.21.15.178:0/1663485061 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c0b4570 0x7f011c139540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 --2- 172.21.15.178:0/1663485061 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f011c0b0ed0 0x7f011c142cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 -- 172.21.15.178:0/1663485061 >> 172.21.15.178:0/1663485061 conn(0x7f011c065d50 msgr2=0x7f011c066130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:35.699 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 -- 172.21.15.178:0/1663485061 shutdown_connections 2023-11-02T08:35:35.699 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 -- 172.21.15.178:0/1663485061 wait complete. 2023-11-02T08:35:35.699 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.692+0000 7f0122fab640 1 librados: shutdown 2023-11-02T08:35:35.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new cf07e867-481c-4d45-945a-88e8e08d186d 4 2023-11-02T08:35:35.809 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.807+0000 7f9b15392640 10 monclient: get_monmap_and_config 2023-11-02T08:35:35.810 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.807+0000 7f9b15392640 10 monclient: build_initial_monmap 2023-11-02T08:35:35.810 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.807+0000 7f9b15392640 10 monclient: monmap: 2023-11-02T08:35:35.810 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:35.813 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.813 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:35:35.807538+0000 2023-11-02T08:35:35.813 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:35:35.807538+0000 2023-11-02T08:35:35.814 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:35.815 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.816 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:35.816 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:35.816 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:35.816 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:35.816 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:35.817 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 1 Processor -- start 2023-11-02T08:35:35.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 1 -- start start 2023-11-02T08:35:35.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 10 monclient: init 2023-11-02T08:35:35.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:35.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-11-02T08:35:35.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b100b5b20 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:35.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9b1006bbe0 0x7f9b1013e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1006bbe0 addr v2:172.21.15.178:3300/0 2023-11-02T08:35:35.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b1006c000 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:35.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1006b750 con 0x7f9b100b5b20 2023-11-02T08:35:35.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b100af590 con 0x7f9b1006c000 2023-11-02T08:35:35.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b100b5f00 con 0x7f9b1006bbe0 2023-11-02T08:35:35.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:35.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.808+0000 7f9b15392640 10 monclient(hunting): authenticate will time out at 1288.134786s 2023-11-02T08:35:35.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.809+0000 7f9b0e7fc640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9b1006bbe0 0x7f9b1013e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.809+0000 7f9b0effd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f9b100b5b20 0x7f9b101417e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.178:49970/0 (socket says 172.21.15.178:49970) 2023-11-02T08:35:35.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.809+0000 7f9b0effd640 1 -- 172.21.15.178:0/1281798482 learned_addr learned my addr 172.21.15.178:0/1281798482 (peer_addr_for_me v1:172.21.15.178:0/0) 2023-11-02T08:35:35.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.809+0000 7f9b0e7fc640 10 monclient(hunting): get_auth_request con 0x7f9b1006bbe0 auth_method 0 2023-11-02T08:35:35.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.809+0000 7f9b0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.809+0000 7f9b0e7fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.809+0000 7f9b0e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.810+0000 7f9b0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.810+0000 7f9b0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.810+0000 7f9b0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0dffb640 1 -- 172.21.15.178:0/1281798482 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651129802 0 0) 0x7f9b1006b750 con 0x7f9b100b5b20 2023-11-02T08:35:35.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0dffb640 10 monclient(hunting): _negotiate 2023-11-02T08:35:35.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0dffb640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0dffb640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0dffb640 10 monclient(hunting): my global_id is 4131 2023-11-02T08:35:35.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0dffb640 1 -- 172.21.15.178:0/1281798482 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af8003520 con 0x7f9b100b5b20 2023-11-02T08:35:35.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0e7fc640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2023-11-02T08:35:35.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0e7fc640 1 -- 172.21.15.178:0/1281798482 >> v1:172.21.15.178:6789/0 conn(0x7f9b1006c000 legacy=0x7f9b1013eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0e7fc640 1 -- 172.21.15.178:0/1281798482 >> v1:172.21.15.151:6789/0 conn(0x7f9b100b5b20 legacy=0x7f9b101417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0e7fc640 10 monclient: _finish_hunting 0 2023-11-02T08:35:35.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0e7fc640 1 monclient: found mon.noname-c 2023-11-02T08:35:35.827 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:35.827 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:35:35.827 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.812+0000 7f9b0e7fc640 1 -- 172.21.15.178:0/1281798482 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b100abf30 con 0x7f9b1006bbe0 2023-11-02T08:35:35.827 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.817+0000 7f9b0e7fc640 1 --2- 172.21.15.178:0/1281798482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b1006bbe0 0x7f9b1013e5c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9b000097a0 tx=0x7f9b0002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=385e2c327bd75b37 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:35.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 1 -- 172.21.15.178:0/1281798482 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00031830 con 0x7f9b1006bbe0 2023-11-02T08:35:35.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.833 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:35:35.833 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 10 monclient: dump: 2023-11-02T08:35:35.833 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.833 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.833 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.834 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.834 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.834 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.834 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.834 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.834 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.835 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 10 monclient: _finish_auth 0 2023-11-02T08:35:35.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 10 monclient: _check_auth_tickets 2023-11-02T08:35:35.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:35.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 1 -- 172.21.15.178:0/1281798482 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b00031990 con 0x7f9b1006bbe0 2023-11-02T08:35:35.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b15392640 5 monclient: authenticate success, global_id 4138 2023-11-02T08:35:35.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b15392640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:35:35.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:35.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b15392640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:35.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b15392640 4 set_mon_vals no callback set 2023-11-02T08:35:35.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 1 -- 172.21.15.178:0/1281798482 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00031d20 con 0x7f9b1006bbe0 2023-11-02T08:35:35.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:35.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.821+0000 7f9b0dffb640 10 monclient: dump: 2023-11-02T08:35:35.838 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.838 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.839 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.839 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.839 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.839 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.839 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.840 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.840 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.840 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.822+0000 7f9b15392640 10 monclient: shutdown 2023-11-02T08:35:35.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.822+0000 7f9b15392640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:35.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.822+0000 7f9b15392640 1 -- 172.21.15.178:0/1281798482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b1006bbe0 msgr2=0x7f9b1013e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.822+0000 7f9b15392640 1 --2- 172.21.15.178:0/1281798482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b1006bbe0 0x7f9b1013e5c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9b000097a0 tx=0x7f9b0002b8d0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.822+0000 7f9b15392640 1 -- 172.21.15.178:0/1281798482 shutdown_connections 2023-11-02T08:35:35.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.822+0000 7f9b15392640 1 --2- 172.21.15.178:0/1281798482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b1006bbe0 0x7f9b1013e5c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.822+0000 7f9b15392640 1 -- 172.21.15.178:0/1281798482 >> 172.21.15.178:0/1281798482 conn(0x7f9b10065f90 msgr2=0x7f9b100b51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:35.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.822+0000 7f9b15392640 1 -- 172.21.15.178:0/1281798482 shutdown_connections 2023-11-02T08:35:35.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.822+0000 7f9b15392640 1 -- 172.21.15.178:0/1281798482 wait complete. 2023-11-02T08:35:35.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.823+0000 7f9b15392640 10 monclient: build_initial_monmap 2023-11-02T08:35:35.843 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.823+0000 7f9b15392640 10 monclient: monmap: 2023-11-02T08:35:35.843 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:35.843 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:35.843 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2023-11-02T08:35:35.843 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2023-11-02T08:35:35.844 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:35.844 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.844 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:35.844 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:35.845 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:35.845 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.823+0000 7f9b15392640 1 librados: starting msgr at 2023-11-02T08:35:35.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 librados: starting objecter 2023-11-02T08:35:35.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 Processor -- start 2023-11-02T08:35:35.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 -- start start 2023-11-02T08:35:35.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 librados: setting wanted keys 2023-11-02T08:35:35.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 librados: calling monclient init 2023-11-02T08:35:35.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient: init 2023-11-02T08:35:35.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:35.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:35:35.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9b1006bbe0 0x7f9b10070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1006bbe0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:35.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b1006c000 0x7f9b100aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient(hunting): picked mon.noname-b con 0x7f9b1006c000 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:35.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9b100b5b20 0x7f9b10138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:35.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient(hunting): picked mon.noname-c con 0x7f9b100b5b20 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:35.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b100ab3f0 con 0x7f9b1006bbe0 2023-11-02T08:35:35.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b100ab110 con 0x7f9b1006c000 2023-11-02T08:35:35.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:35.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b100aa210 con 0x7f9b100b5b20 2023-11-02T08:35:35.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:35.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.824+0000 7f9b15392640 10 monclient(hunting): authenticate will time out at 1288.150927s 2023-11-02T08:35:35.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0effd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9b1006bbe0 0x7f9b10070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0e7fc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b1006c000 0x7f9b100aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0f7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9b100b5b20 0x7f9b10138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:35.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0effd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9b1006bbe0 0x7f9b10070860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.178:35834/0 (socket says 172.21.15.178:35834) 2023-11-02T08:35:35.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0effd640 1 -- 172.21.15.178:0/2534603227 learned_addr learned my addr 172.21.15.178:0/2534603227 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-02T08:35:35.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0effd640 10 monclient(hunting): get_auth_request con 0x7f9b1006bbe0 auth_method 0 2023-11-02T08:35:35.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0effd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0effd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0f7fe640 10 monclient(hunting): get_auth_request con 0x7f9b100b5b20 auth_method 0 2023-11-02T08:35:35.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.825+0000 7f9b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0e7fc640 10 monclient(hunting): get_auth_request con 0x7f9b1006c000 auth_method 0 2023-11-02T08:35:35.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:35.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0e7fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:35.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:35.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.856 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.856 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.856 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.856 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:35.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:35.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.826+0000 7f9b0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:35.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 10 monclient(hunting): handle_auth_done global_id 4121 payload 824 2023-11-02T08:35:35.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 1 -- 172.21.15.178:0/2534603227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b1006c000 msgr2=0x7f9b100aabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 1 --2- 172.21.15.178:0/2534603227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b1006c000 0x7f9b100aabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 1 -- 172.21.15.178:0/2534603227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9b1006bbe0 msgr2=0x7f9b10070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:35.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 1 --2- 172.21.15.178:0/2534603227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9b1006bbe0 0x7f9b10070860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:35.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 10 monclient: _finish_hunting 0 2023-11-02T08:35:35.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 1 monclient: found mon.noname-c 2023-11-02T08:35:35.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:35.859 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:35:35.859 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.829+0000 7f9b0f7fe640 1 -- 172.21.15.178:0/2534603227 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b00009400 con 0x7f9b100b5b20 2023-11-02T08:35:35.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.831+0000 7f9b0f7fe640 1 --2- 172.21.15.178:0/2534603227 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9b100b5b20 0x7f9b10138790 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c700 tx=0x7f9b0800cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:35.862 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.831+0000 7f9af77fe640 10 client.?.objecter ms_handle_connect 0x7f9b100b5b20 2023-11-02T08:35:35.862 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.831+0000 7f9af77fe640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:35.862 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 1 -- 172.21.15.178:0/2534603227 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08003b70 con 0x7f9b100b5b20 2023-11-02T08:35:35.863 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.863 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:35:35.863 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 10 monclient: dump: 2023-11-02T08:35:35.863 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.863 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.864 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.864 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.864 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.864 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.865 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.865 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.865 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.866 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.866 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 10 monclient: _finish_auth 0 2023-11-02T08:35:35.866 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 10 monclient: _check_auth_tickets 2023-11-02T08:35:35.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:35.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 1 -- 172.21.15.178:0/2534603227 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b08003cd0 con 0x7f9b100b5b20 2023-11-02T08:35:35.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:35.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 1 -- 172.21.15.178:0/2534603227 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08005310 con 0x7f9b100b5b20 2023-11-02T08:35:35.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9b15392640 5 monclient: authenticate success, global_id 4121 2023-11-02T08:35:35.868 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9b15392640 10 monclient: _renew_subs 2023-11-02T08:35:35.868 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9b15392640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:35:35.868 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9b15392640 1 -- 172.21.15.178:0/2534603227 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b100aa430 con 0x7f9b100b5b20 2023-11-02T08:35:35.868 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:35.868 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:35:35.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9b0d7fa640 4 set_mon_vals no callback set 2023-11-02T08:35:35.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.834+0000 7f9af77fe640 10 monclient: dump: 2023-11-02T08:35:35.869 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:35.869 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:35.869 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.870 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:35.870 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:35.870 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:35.870 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:35.870 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:35.871 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:35.871 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:35.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.835+0000 7f9b15392640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:35.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.835+0000 7f9b15392640 10 monclient: _renew_subs 2023-11-02T08:35:35.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.835+0000 7f9b15392640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:35:35.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.835+0000 7f9b15392640 1 -- 172.21.15.178:0/2534603227 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b10138eb0 con 0x7f9b100b5b20 2023-11-02T08:35:35.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.835+0000 7f9b15392640 1 librados: init done 2023-11-02T08:35:35.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.835+0000 7f9b15392640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:35.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.835+0000 7f9b15392640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:35.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.835+0000 7f9b15392640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:35:35.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.835+0000 7f9b15392640 1 -- 172.21.15.178:0/2534603227 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b10067920 con 0x7f9b100b5b20 2023-11-02T08:35:35.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.838+0000 7f9af77fe640 1 -- 172.21.15.178:0/2534603227 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9b0800a640 con 0x7f9b100b5b20 2023-11-02T08:35:35.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.838+0000 7f9af77fe640 1 -- 172.21.15.178:0/2534603227 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f9b08025940 con 0x7f9b100b5b20 2023-11-02T08:35:35.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.838+0000 7f9af77fe640 10 client.4121.objecter ms_dispatch 0x7f9b10000b90 osd_map(6..6 src has 1..6) v4 2023-11-02T08:35:35.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.838+0000 7f9af77fe640 3 client.4121.objecter handle_osd_map got epochs [6,6] > 0 2023-11-02T08:35:35.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.838+0000 7f9af77fe640 3 client.4121.objecter handle_osd_map decoding full epoch 6 2023-11-02T08:35:35.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:35.838+0000 7f9af77fe640 20 client.4121.objecter dump_active .. 0 homeless 2023-11-02T08:35:36.023 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.020+0000 7f9af77fe640 1 -- 172.21.15.178:0/2534603227 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f9b08005770 con 0x7f9b100b5b20 2023-11-02T08:35:36.023 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.020+0000 7f9af77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:36.023 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.020+0000 7f9af77fe640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:36.061 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:36.059+0000 7fb1fc810040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-02T08:35:36.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.092+0000 7f9b15392640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cf07e867-481c-4d45-945a-88e8e08d186d", "id": 4}] 2023-11-02T08:35:36.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.092+0000 7f9b15392640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cf07e867-481c-4d45-945a-88e8e08d186d", "id": 4}] 2023-11-02T08:35:36.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.092+0000 7f9b15392640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:35:36.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.092+0000 7f9b15392640 1 -- 172.21.15.178:0/2534603227 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cf07e867-481c-4d45-945a-88e8e08d186d", "id": 4} v 0) v1 -- 0x7f9b1006b2f0 con 0x7f9b100b5b20 2023-11-02T08:35:36.171 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:36.167+0000 7f0836c1f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-02T08:35:36.204 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.201+0000 7f9af77fe640 1 -- 172.21.15.178:0/2534603227 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cf07e867-481c-4d45-945a-88e8e08d186d", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9b08025ce0 con 0x7f9b100b5b20 2023-11-02T08:35:36.204 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9af77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cf07e867-481c-4d45-945a-88e8e08d186d", "id": 4}] 2023-11-02T08:35:36.205 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9af77fe640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:36.205 INFO:teuthology.orchestra.run.smithi178.stdout:4 2023-11-02T08:35:36.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9b15392640 10 librados: watch_flush enter 2023-11-02T08:35:36.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9b15392640 10 librados: watch_flush exit 2023-11-02T08:35:36.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9b15392640 20 client.4121.objecter shutdown clearing up homeless session... 2023-11-02T08:35:36.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9b15392640 10 client.4121.objecter successfully canceled tick 2023-11-02T08:35:36.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9b15392640 10 monclient: shutdown 2023-11-02T08:35:36.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9b15392640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:36.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9b15392640 1 -- 172.21.15.178:0/2534603227 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9b100b5b20 msgr2=0x7f9b10138790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.202+0000 7f9b15392640 1 --2- 172.21.15.178:0/2534603227 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9b100b5b20 0x7f9b10138790 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c700 tx=0x7f9b0800cbc0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.203+0000 7f9b15392640 1 -- 172.21.15.178:0/2534603227 shutdown_connections 2023-11-02T08:35:36.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.203+0000 7f9b15392640 1 --2- 172.21.15.178:0/2534603227 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9b100b5b20 0x7f9b10138790 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.203+0000 7f9b15392640 1 --2- 172.21.15.178:0/2534603227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b1006c000 0x7f9b100aabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.203+0000 7f9b15392640 1 --2- 172.21.15.178:0/2534603227 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9b1006bbe0 0x7f9b10070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.203+0000 7f9b15392640 1 -- 172.21.15.178:0/2534603227 >> 172.21.15.178:0/2534603227 conn(0x7f9b10065f90 msgr2=0x7f9b100b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:36.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.203+0000 7f9b15392640 1 -- 172.21.15.178:0/2534603227 shutdown_connections 2023-11-02T08:35:36.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.203+0000 7f9b15392640 1 -- 172.21.15.178:0/2534603227 wait complete. 2023-11-02T08:35:36.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.203+0000 7f9b15392640 1 librados: shutdown 2023-11-02T08:35:36.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new da85ceeb-7ee7-44cb-8460-4d0aad9c6e70 5 2023-11-02T08:35:36.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.312+0000 7f72285a4640 10 monclient: get_monmap_and_config 2023-11-02T08:35:36.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.312+0000 7f72285a4640 10 monclient: build_initial_monmap 2023-11-02T08:35:36.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.312+0000 7f72285a4640 10 monclient: monmap: 2023-11-02T08:35:36.315 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:36.316 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.316 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:35:36.312301+0000 2023-11-02T08:35:36.316 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:35:36.312301+0000 2023-11-02T08:35:36.316 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:36.316 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.317 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:36.317 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:36.317 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:36.317 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:36.317 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:36.318 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.318 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.312+0000 7f72285a4640 1 Processor -- start 2023-11-02T08:35:36.318 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.312+0000 7f72285a4640 1 -- start start 2023-11-02T08:35:36.318 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.312+0000 7f72285a4640 10 monclient: init 2023-11-02T08:35:36.321 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:36.321 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-11-02T08:35:36.322 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72200b4570 0x7f7220144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:36.322 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 10 monclient(hunting): picked mon.noname-b con 0x7f72200b4570 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:36.322 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72200b0ed0 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:36.322 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f722013e520 0x7f722013e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:36.323 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 10 monclient(hunting): picked mon.noname-a con 0x7f722013e520 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:36.323 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72200a9780 con 0x7f72200b0ed0 2023-11-02T08:35:36.323 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:36.323 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72200b4950 con 0x7f722013e520 2023-11-02T08:35:36.323 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:36.324 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72200aa1d0 con 0x7f72200b4570 2023-11-02T08:35:36.324 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:36.324 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.313+0000 7f72285a4640 10 monclient(hunting): authenticate will time out at 1288.639591s 2023-11-02T08:35:36.324 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7225b3f640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f72200b0ed0 0x7f72200a9070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:40022/0 (socket says 172.21.15.178:40022) 2023-11-02T08:35:36.324 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7225b3f640 1 -- 172.21.15.178:0/444634416 learned_addr learned my addr 172.21.15.178:0/444634416 (peer_addr_for_me v1:172.21.15.178:0/0) 2023-11-02T08:35:36.325 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226b41640 1 --2- 172.21.15.178:0/444634416 >> v2:172.21.15.151:3300/0 conn(0x7f722013e520 0x7f722013e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:36.325 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226340640 1 --2- 172.21.15.178:0/444634416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72200b4570 0x7f7220144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:36.325 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226b41640 10 monclient(hunting): get_auth_request con 0x7f722013e520 auth_method 0 2023-11-02T08:35:36.325 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226b41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:36.326 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226b41640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:36.326 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226b41640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:36.326 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226340640 10 monclient(hunting): get_auth_request con 0x7f72200b4570 auth_method 0 2023-11-02T08:35:36.326 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226340640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:36.326 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226340640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:36.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.314+0000 7f7226340640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:36.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.315+0000 7f7226b41640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:36.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.315+0000 7f7226b41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:36.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.315+0000 7f7226b41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:36.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.315+0000 7f7226340640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:36.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.315+0000 7f7226340640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:36.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.315+0000 7f7226340640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:36.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.316+0000 7f722533e640 1 -- 172.21.15.178:0/444634416 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501039244 0 0) 0x7f72200a9780 con 0x7f72200b0ed0 2023-11-02T08:35:36.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.316+0000 7f722533e640 10 monclient(hunting): _negotiate 2023-11-02T08:35:36.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.316+0000 7f722533e640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:36.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.316+0000 7f722533e640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:36.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.316+0000 7f722533e640 10 monclient(hunting): my global_id is 4147 2023-11-02T08:35:36.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.316+0000 7f722533e640 1 -- 172.21.15.178:0/444634416 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f720c003520 con 0x7f72200b0ed0 2023-11-02T08:35:36.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.317+0000 7f7226b41640 10 monclient(hunting): handle_auth_done global_id 4137 payload 274 2023-11-02T08:35:36.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.317+0000 7f7226b41640 1 -- 172.21.15.178:0/444634416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72200b4570 msgr2=0x7f7220144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:36.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.317+0000 7f7226b41640 1 --2- 172.21.15.178:0/444634416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72200b4570 0x7f7220144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.317+0000 7f7226b41640 1 -- 172.21.15.178:0/444634416 >> v1:172.21.15.178:6789/0 conn(0x7f72200b0ed0 legacy=0x7f72200a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.317+0000 7f7226b41640 10 monclient: _finish_hunting 0 2023-11-02T08:35:36.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.317+0000 7f7226b41640 1 monclient: found mon.noname-a 2023-11-02T08:35:36.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.317+0000 7f7226b41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:36.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.317+0000 7f7226b41640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:36.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.317+0000 7f7226b41640 1 -- 172.21.15.178:0/444634416 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72200aa4b0 con 0x7f722013e520 2023-11-02T08:35:36.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.319+0000 7f7226b41640 1 --2- 172.21.15.178:0/444634416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f722013e520 0x7f722013e990 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7218009d90 tx=0x7f721802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d749b096f0c762d server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:36.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 1 -- 172.21.15.178:0/444634416 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721802cb30 con 0x7f722013e520 2023-11-02T08:35:36.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:36.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:36.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 10 monclient: dump: 2023-11-02T08:35:36.337 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:36.337 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.337 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.337 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.337 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:36.338 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.338 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:36.338 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:36.338 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:36.338 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 10 monclient: _finish_auth 0 2023-11-02T08:35:36.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 10 monclient: _check_auth_tickets 2023-11-02T08:35:36.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:36.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 1 -- 172.21.15.178:0/444634416 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721802cc90 con 0x7f722013e520 2023-11-02T08:35:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 1 -- 172.21.15.178:0/444634416 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218034440 con 0x7f722013e520 2023-11-02T08:35:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:36.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f722533e640 10 monclient: dump: 2023-11-02T08:35:36.341 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:36.341 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.341 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.341 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.341 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:36.342 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.342 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:36.342 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:36.342 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:36.342 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f72285a4640 5 monclient: authenticate success, global_id 4137 2023-11-02T08:35:36.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f72285a4640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:36.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.323+0000 7f72285a4640 4 set_mon_vals no callback set 2023-11-02T08:35:36.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 10 monclient: shutdown 2023-11-02T08:35:36.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:36.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 1 -- 172.21.15.178:0/444634416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f722013e520 msgr2=0x7f722013e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 1 --2- 172.21.15.178:0/444634416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f722013e520 0x7f722013e990 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7218009d90 tx=0x7f721802bd80 comp rx=0 tx=0).stop 2023-11-02T08:35:36.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 1 -- 172.21.15.178:0/444634416 shutdown_connections 2023-11-02T08:35:36.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 1 --2- 172.21.15.178:0/444634416 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f722013e520 0x7f722013e990 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 1 --2- 172.21.15.178:0/444634416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72200b4570 0x7f7220144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 1 -- 172.21.15.178:0/444634416 >> 172.21.15.178:0/444634416 conn(0x7f7220065d50 msgr2=0x7f7220066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:36.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 1 -- 172.21.15.178:0/444634416 shutdown_connections 2023-11-02T08:35:36.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.324+0000 7f72285a4640 1 -- 172.21.15.178:0/444634416 wait complete. 2023-11-02T08:35:36.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient: build_initial_monmap 2023-11-02T08:35:36.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient: monmap: 2023-11-02T08:35:36.346 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:36.346 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:36.346 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2023-11-02T08:35:36.346 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2023-11-02T08:35:36.347 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:36.347 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.347 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:36.347 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:36.347 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:36.348 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.348 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 librados: starting msgr at 2023-11-02T08:35:36.348 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 librados: starting objecter 2023-11-02T08:35:36.348 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 Processor -- start 2023-11-02T08:35:36.348 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 -- start start 2023-11-02T08:35:36.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 librados: setting wanted keys 2023-11-02T08:35:36.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 librados: calling monclient init 2023-11-02T08:35:36.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient: init 2023-11-02T08:35:36.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:36.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:35:36.350 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72200b0ed0 0x7f72200ac720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:36.350 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient(hunting): picked mon.noname-b con 0x7f72200b0ed0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:36.350 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72200b4570 0x7f72201d9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:36.350 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient(hunting): picked mon.noname-a con 0x7f72200b4570 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:36.351 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f722013e520 0x7f72201da280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:36.351 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient(hunting): picked mon.noname-c con 0x7f722013e520 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:36.351 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:36.351 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72201d98b0 con 0x7f72200b4570 2023-11-02T08:35:36.351 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:36.351 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72200aadb0 con 0x7f72200b0ed0 2023-11-02T08:35:36.352 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:36.352 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72200aaf70 con 0x7f722013e520 2023-11-02T08:35:36.352 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:36.352 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.325+0000 7f72285a4640 10 monclient(hunting): authenticate will time out at 1288.652031s 2023-11-02T08:35:36.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7226340640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72200b0ed0 0x7f72200ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:36.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7225b3f640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72200b4570 0x7f72201d9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:36.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7226b41640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f722013e520 0x7f72201da280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:36.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7225b3f640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72200b4570 0x7f72201d9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.178:35852/0 (socket says 172.21.15.178:35852) 2023-11-02T08:35:36.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7225b3f640 1 -- 172.21.15.178:0/2969226483 learned_addr learned my addr 172.21.15.178:0/2969226483 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-02T08:35:36.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7225b3f640 10 monclient(hunting): get_auth_request con 0x7f72200b4570 auth_method 0 2023-11-02T08:35:36.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7225b3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:36.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7225b3f640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:36.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7225b3f640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:36.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7226340640 10 monclient(hunting): get_auth_request con 0x7f72200b0ed0 auth_method 0 2023-11-02T08:35:36.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7226340640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:36.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7226340640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:36.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.326+0000 7f7226340640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:36.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226b41640 10 monclient(hunting): get_auth_request con 0x7f722013e520 auth_method 0 2023-11-02T08:35:36.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226b41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:36.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226b41640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:36.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226b41640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:36.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7225b3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:36.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7225b3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:36.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7225b3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:36.357 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226340640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:36.357 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226340640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:36.357 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226340640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:36.357 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226b41640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:36.357 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226b41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:36.358 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.327+0000 7f7226b41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:36.358 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 10 monclient(hunting): handle_auth_done global_id 4140 payload 824 2023-11-02T08:35:36.359 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 1 -- 172.21.15.178:0/2969226483 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f722013e520 msgr2=0x7f72201da280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.359 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 1 --2- 172.21.15.178:0/2969226483 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f722013e520 0x7f72201da280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.359 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 1 -- 172.21.15.178:0/2969226483 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72200b0ed0 msgr2=0x7f72200ac720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 1 --2- 172.21.15.178:0/2969226483 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72200b0ed0 0x7f72200ac720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 10 monclient: _finish_hunting 0 2023-11-02T08:35:36.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 1 monclient: found mon.noname-a 2023-11-02T08:35:36.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:36.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:36.361 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.331+0000 7f7225b3f640 1 -- 172.21.15.178:0/2969226483 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7218009a70 con 0x7f72200b4570 2023-11-02T08:35:36.363 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.334+0000 7f7225b3f640 1 --2- 172.21.15.178:0/2969226483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72200b4570 0x7f72201d9d40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f721000c4d0 tx=0x7f721000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:36.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.334+0000 7f7216ffd640 10 client.?.objecter ms_handle_connect 0x7f72200b4570 2023-11-02T08:35:36.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.334+0000 7f7216ffd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:36.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 1 -- 172.21.15.178:0/2969226483 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210007480 con 0x7f72200b4570 2023-11-02T08:35:36.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:36.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:36.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 10 monclient: dump: 2023-11-02T08:35:36.365 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:36.366 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.366 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.366 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.366 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:36.366 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.367 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:36.367 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:36.367 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:36.367 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.367 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 10 monclient: _finish_auth 0 2023-11-02T08:35:36.368 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 10 monclient: _check_auth_tickets 2023-11-02T08:35:36.368 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:36.368 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 1 -- 172.21.15.178:0/2969226483 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72100075e0 con 0x7f72200b4570 2023-11-02T08:35:36.368 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:36.368 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 1 -- 172.21.15.178:0/2969226483 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72100059c0 con 0x7f72200b4570 2023-11-02T08:35:36.369 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f72285a4640 5 monclient: authenticate success, global_id 4140 2023-11-02T08:35:36.369 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f72285a4640 10 monclient: _renew_subs 2023-11-02T08:35:36.369 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f72285a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:36.369 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f72285a4640 1 -- 172.21.15.178:0/2969226483 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72200ac150 con 0x7f72200b4570 2023-11-02T08:35:36.369 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7224b3d640 4 set_mon_vals no callback set 2023-11-02T08:35:36.370 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:36.370 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:36.370 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f7216ffd640 10 monclient: dump: 2023-11-02T08:35:36.370 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:36.371 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.371 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.371 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.371 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:36.372 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.372 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:36.372 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:36.372 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:36.373 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f72285a4640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:36.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f72285a4640 10 monclient: _renew_subs 2023-11-02T08:35:36.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f72285a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:36.374 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f72285a4640 1 -- 172.21.15.178:0/2969226483 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72201420d0 con 0x7f72200b4570 2023-11-02T08:35:36.374 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.337+0000 7f72285a4640 1 librados: init done 2023-11-02T08:35:36.374 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.338+0000 7f72285a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:36.375 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.338+0000 7f72285a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:36.375 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.338+0000 7f72285a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:36.375 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.338+0000 7f72285a4640 1 -- 172.21.15.178:0/2969226483 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7220067920 con 0x7f72200b4570 2023-11-02T08:35:36.375 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.339+0000 7f7216ffd640 1 -- 172.21.15.178:0/2969226483 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f721001c440 con 0x7f72200b4570 2023-11-02T08:35:36.376 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:36.344+0000 7fb1fc810040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-02T08:35:36.398 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:36.395+0000 7fb1fc810040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-02T08:35:36.444 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.442+0000 7f7216ffd640 1 -- 172.21.15.178:0/2969226483 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f721001cc70 con 0x7f72200b4570 2023-11-02T08:35:36.445 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.442+0000 7f7216ffd640 10 client.4140.objecter ms_dispatch 0x7f7220000b90 osd_map(7..7 src has 1..7) v4 2023-11-02T08:35:36.445 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.442+0000 7f7216ffd640 3 client.4140.objecter handle_osd_map got epochs [7,7] > 0 2023-11-02T08:35:36.445 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.442+0000 7f7216ffd640 3 client.4140.objecter handle_osd_map decoding full epoch 7 2023-11-02T08:35:36.445 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.442+0000 7f7216ffd640 20 client.4140.objecter dump_active .. 0 homeless 2023-11-02T08:35:36.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.442+0000 7f7216ffd640 1 -- 172.21.15.178:0/2969226483 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f7210017070 con 0x7f72200b4570 2023-11-02T08:35:36.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.442+0000 7f7216ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:36.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.442+0000 7f7216ffd640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:36.456 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:36.453+0000 7f0836c1f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-02T08:35:36.499 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:36.496+0000 7fb1fc810040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-02T08:35:36.507 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:36.504+0000 7f0836c1f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-02T08:35:36.515 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.513+0000 7f72285a4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "da85ceeb-7ee7-44cb-8460-4d0aad9c6e70", "id": 5}] 2023-11-02T08:35:36.516 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.513+0000 7f72285a4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "da85ceeb-7ee7-44cb-8460-4d0aad9c6e70", "id": 5}] 2023-11-02T08:35:36.516 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.513+0000 7f72285a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:36.516 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.513+0000 7f72285a4640 1 -- 172.21.15.178:0/2969226483 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "da85ceeb-7ee7-44cb-8460-4d0aad9c6e70", "id": 5} v 0) v1 -- 0x7f72200a8bb0 con 0x7f72200b4570 2023-11-02T08:35:36.610 INFO:tasks.ceph.mgr.y.smithi151.stderr:2023-11-02T08:35:36.607+0000 7f0836c1f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-02T08:35:36.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.702+0000 7f7216ffd640 1 -- 172.21.15.178:0/2969226483 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "da85ceeb-7ee7-44cb-8460-4d0aad9c6e70", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f721001c8e0 con 0x7f72200b4570 2023-11-02T08:35:36.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.702+0000 7f7216ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "da85ceeb-7ee7-44cb-8460-4d0aad9c6e70", "id": 5}] 2023-11-02T08:35:36.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.702+0000 7f7216ffd640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:36.707 INFO:teuthology.orchestra.run.smithi178.stdout:5 2023-11-02T08:35:36.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 10 librados: watch_flush enter 2023-11-02T08:35:36.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 10 librados: watch_flush exit 2023-11-02T08:35:36.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 20 client.4140.objecter shutdown clearing up homeless session... 2023-11-02T08:35:36.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 10 client.4140.objecter successfully canceled tick 2023-11-02T08:35:36.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 10 monclient: shutdown 2023-11-02T08:35:36.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:36.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 -- 172.21.15.178:0/2969226483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72200b4570 msgr2=0x7f72201d9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 --2- 172.21.15.178:0/2969226483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72200b4570 0x7f72201d9d40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f721000c4d0 tx=0x7f721000c990 comp rx=0 tx=0).stop 2023-11-02T08:35:36.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 -- 172.21.15.178:0/2969226483 shutdown_connections 2023-11-02T08:35:36.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 --2- 172.21.15.178:0/2969226483 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f722013e520 0x7f72201da280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 --2- 172.21.15.178:0/2969226483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72200b4570 0x7f72201d9d40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 --2- 172.21.15.178:0/2969226483 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72200b0ed0 0x7f72200ac720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 -- 172.21.15.178:0/2969226483 >> 172.21.15.178:0/2969226483 conn(0x7f7220065d50 msgr2=0x7f722013d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:36.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 -- 172.21.15.178:0/2969226483 shutdown_connections 2023-11-02T08:35:36.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 -- 172.21.15.178:0/2969226483 wait complete. 2023-11-02T08:35:36.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.703+0000 7f72285a4640 1 librados: shutdown 2023-11-02T08:35:36.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new bfab5b18-727d-4c9f-a9f0-30d624e82282 6 2023-11-02T08:35:36.815 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.811+0000 7f2be8cc4640 10 monclient: get_monmap_and_config 2023-11-02T08:35:36.815 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.811+0000 7f2be8cc4640 10 monclient: build_initial_monmap 2023-11-02T08:35:36.815 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.811+0000 7f2be8cc4640 10 monclient: monmap: 2023-11-02T08:35:36.815 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:36.816 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.816 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:35:36.812912+0000 2023-11-02T08:35:36.816 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:35:36.812912+0000 2023-11-02T08:35:36.816 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:36.816 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.817 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:36.817 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:36.817 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:36.817 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:36.817 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:36.817 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.812+0000 7f2be8cc4640 1 Processor -- start 2023-11-02T08:35:36.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.812+0000 7f2be8cc4640 1 -- start start 2023-11-02T08:35:36.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.812+0000 7f2be8cc4640 10 monclient: init 2023-11-02T08:35:36.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:36.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-11-02T08:35:36.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2be406fe90 0x7f2be4070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:36.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 10 monclient(hunting): picked mon.noname-c con 0x7f2be406fe90 addr v2:172.21.15.178:3300/0 2023-11-02T08:35:36.821 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2be413c3d0 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:36.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2be413f5f0 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:36.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be413cf20 con 0x7f2be413f5f0 2023-11-02T08:35:36.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be4071110 con 0x7f2be413c3d0 2023-11-02T08:35:36.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:36.822 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be4071e00 con 0x7f2be406fe90 2023-11-02T08:35:36.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:36.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be8cc4640 10 monclient(hunting): authenticate will time out at 1289.140326s 2023-11-02T08:35:36.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.813+0000 7f2be259c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2be406fe90 0x7f2be4070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:36.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.814+0000 7f2be259c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2be406fe90 0x7f2be4070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:42656/0 (socket says 172.21.15.178:42656) 2023-11-02T08:35:36.823 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.814+0000 7f2be259c640 1 -- 172.21.15.178:0/1140491774 learned_addr learned my addr 172.21.15.178:0/1140491774 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-02T08:35:36.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.814+0000 7f2be259c640 10 monclient(hunting): get_auth_request con 0x7f2be406fe90 auth_method 0 2023-11-02T08:35:36.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.814+0000 7f2be259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:36.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.814+0000 7f2be259c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:36.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.814+0000 7f2be259c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:36.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.814+0000 7f2be259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:36.824 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.814+0000 7f2be259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:36.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.814+0000 7f2be259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:36.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.816+0000 7f2be259c640 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2023-11-02T08:35:36.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.816+0000 7f2be259c640 1 -- 172.21.15.178:0/1140491774 >> v1:172.21.15.178:6789/0 conn(0x7f2be413c3d0 legacy=0x7f2be413c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.816+0000 7f2be259c640 1 -- 172.21.15.178:0/1140491774 >> v1:172.21.15.151:6789/0 conn(0x7f2be413f5f0 legacy=0x7f2be413fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:36.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.816+0000 7f2be259c640 10 monclient: _finish_hunting 0 2023-11-02T08:35:36.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.816+0000 7f2be259c640 1 monclient: found mon.noname-c 2023-11-02T08:35:36.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.816+0000 7f2be259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:36.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.816+0000 7f2be259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:35:36.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.816+0000 7f2be259c640 1 -- 172.21.15.178:0/1140491774 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40a8550 con 0x7f2be406fe90 2023-11-02T08:35:36.828 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.820+0000 7f2be259c640 1 --2- 172.21.15.178:0/1140491774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 0x7f2be4070270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009870 tx=0x7f2bd402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5e7f41026f373fe server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:36.828 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 1 -- 172.21.15.178:0/1140491774 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd40317e0 con 0x7f2be406fe90 2023-11-02T08:35:36.828 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:36.828 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:35:36.829 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 10 monclient: dump: 2023-11-02T08:35:36.829 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:36.829 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.829 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.829 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.830 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:36.830 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.830 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:36.830 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:36.830 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:36.831 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.831 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 10 monclient: _finish_auth 0 2023-11-02T08:35:36.831 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 10 monclient: _check_auth_tickets 2023-11-02T08:35:36.831 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:36.831 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 1 -- 172.21.15.178:0/1140491774 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd4031940 con 0x7f2be406fe90 2023-11-02T08:35:36.831 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be8cc4640 5 monclient: authenticate success, global_id 4153 2023-11-02T08:35:36.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be8cc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:35:36.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:36.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be8cc4640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:36.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be8cc4640 4 set_mon_vals no callback set 2023-11-02T08:35:36.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 1 -- 172.21.15.178:0/1140491774 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4031cd0 con 0x7f2be406fe90 2023-11-02T08:35:36.833 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:36.833 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:36.833 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.823+0000 7f2be159a640 10 monclient: dump: 2023-11-02T08:35:36.833 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:36.833 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.834 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.834 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.834 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:36.834 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.834 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:36.835 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:36.835 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:36.835 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.824+0000 7f2be8cc4640 10 monclient: shutdown 2023-11-02T08:35:36.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.824+0000 7f2be8cc4640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:36.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.824+0000 7f2be8cc4640 1 -- 172.21.15.178:0/1140491774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 msgr2=0x7f2be4070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.824+0000 7f2be8cc4640 1 --2- 172.21.15.178:0/1140491774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 0x7f2be4070270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009870 tx=0x7f2bd402b880 comp rx=0 tx=0).stop 2023-11-02T08:35:36.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.824+0000 7f2be8cc4640 1 -- 172.21.15.178:0/1140491774 shutdown_connections 2023-11-02T08:35:36.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.824+0000 7f2be8cc4640 1 --2- 172.21.15.178:0/1140491774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 0x7f2be4070270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.824+0000 7f2be8cc4640 1 -- 172.21.15.178:0/1140491774 >> 172.21.15.178:0/1140491774 conn(0x7f2be40b54e0 msgr2=0x7f2be40b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:36.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.824+0000 7f2be8cc4640 1 -- 172.21.15.178:0/1140491774 shutdown_connections 2023-11-02T08:35:36.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.824+0000 7f2be8cc4640 1 -- 172.21.15.178:0/1140491774 wait complete. 2023-11-02T08:35:36.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.825+0000 7f2be8cc4640 10 monclient: build_initial_monmap 2023-11-02T08:35:36.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.825+0000 7f2be8cc4640 10 monclient: monmap: 2023-11-02T08:35:36.842 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:36.842 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:36.842 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2023-11-02T08:35:36.843 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2023-11-02T08:35:36.843 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:36.843 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.843 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:36.843 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:36.844 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:36.844 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.825+0000 7f2be8cc4640 1 librados: starting msgr at 2023-11-02T08:35:36.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.825+0000 7f2be8cc4640 1 librados: starting objecter 2023-11-02T08:35:36.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.825+0000 7f2be8cc4640 1 Processor -- start 2023-11-02T08:35:36.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 1 -- start start 2023-11-02T08:35:36.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 1 librados: setting wanted keys 2023-11-02T08:35:36.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 1 librados: calling monclient init 2023-11-02T08:35:36.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient: init 2023-11-02T08:35:36.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:36.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:35:36.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 0x7f2be41356c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:36.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient(hunting): picked mon.noname-b con 0x7f2be406fe90 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:36.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2be413c3d0 0x7f2be4137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:36.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient(hunting): picked mon.noname-a con 0x7f2be413c3d0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:36.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2be413f5f0 0x7f2be4141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:36.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient(hunting): picked mon.noname-c con 0x7f2be413f5f0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:36.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:36.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be41375f0 con 0x7f2be413c3d0 2023-11-02T08:35:36.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:36.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be4136640 con 0x7f2be406fe90 2023-11-02T08:35:36.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:36.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be4136920 con 0x7f2be413f5f0 2023-11-02T08:35:36.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:36.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be8cc4640 10 monclient(hunting): authenticate will time out at 1289.153609s 2023-11-02T08:35:36.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.826+0000 7f2be259c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 0x7f2be41356c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:36.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be259c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 0x7f2be41356c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:42666/0 (socket says 172.21.15.178:42666) 2023-11-02T08:35:36.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be2d9d640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2be413f5f0 0x7f2be4141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:36.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be259c640 1 -- 172.21.15.178:0/3723254508 learned_addr learned my addr 172.21.15.178:0/3723254508 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-02T08:35:36.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be259c640 10 monclient(hunting): get_auth_request con 0x7f2be406fe90 auth_method 0 2023-11-02T08:35:36.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:36.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be259c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:36.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be259c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:36.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:36.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:36.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:36.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be2d9d640 10 monclient(hunting): get_auth_request con 0x7f2be413f5f0 auth_method 0 2023-11-02T08:35:36.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:36.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be2d9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:36.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.827+0000 7f2be2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:36.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.828+0000 7f2be2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:36.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.828+0000 7f2be2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:36.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.828+0000 7f2be2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:36.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 10 monclient(hunting): handle_auth_done global_id 4156 payload 824 2023-11-02T08:35:36.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 1 -- 172.21.15.178:0/3723254508 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2be413f5f0 msgr2=0x7f2be4141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 1 --2- 172.21.15.178:0/3723254508 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2be413f5f0 0x7f2be4141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 1 -- 172.21.15.178:0/3723254508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2be413c3d0 msgr2=0x7f2be4137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:36.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 1 --2- 172.21.15.178:0/3723254508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2be413c3d0 0x7f2be4137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:36.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 10 monclient: _finish_hunting 0 2023-11-02T08:35:36.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 1 monclient: found mon.noname-b 2023-11-02T08:35:36.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:36.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:35:36.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.830+0000 7f2be259c640 1 -- 172.21.15.178:0/3723254508 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd40094d0 con 0x7f2be406fe90 2023-11-02T08:35:36.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.832+0000 7f2be259c640 1 --2- 172.21.15.178:0/3723254508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 0x7f2be41356c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2bd40090b0 tx=0x7f2bd40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:36.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.832+0000 7f2bd2ffd640 10 client.?.objecter ms_handle_connect 0x7f2be406fe90 2023-11-02T08:35:36.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.832+0000 7f2bd2ffd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:36.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 1 -- 172.21.15.178:0/3723254508 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4032940 con 0x7f2be406fe90 2023-11-02T08:35:36.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:36.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:35:36.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 10 monclient: dump: 2023-11-02T08:35:36.942 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:36.942 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.942 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.943 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.943 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:36.943 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.943 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:36.943 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:36.943 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:36.944 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.944 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 10 monclient: _finish_auth 0 2023-11-02T08:35:36.944 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 10 monclient: _check_auth_tickets 2023-11-02T08:35:36.944 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:36.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 1 -- 172.21.15.178:0/3723254508 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd4032ac0 con 0x7f2be406fe90 2023-11-02T08:35:36.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be8cc4640 5 monclient: authenticate success, global_id 4156 2023-11-02T08:35:36.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:36.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be8cc4640 10 monclient: _renew_subs 2023-11-02T08:35:36.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 1 -- 172.21.15.178:0/3723254508 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4032e70 con 0x7f2be406fe90 2023-11-02T08:35:36.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be8cc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:36.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be8cc4640 1 -- 172.21.15.178:0/3723254508 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be4136c00 con 0x7f2be406fe90 2023-11-02T08:35:36.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:36.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be0d99640 4 set_mon_vals no callback set 2023-11-02T08:35:36.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:36.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2bd2ffd640 10 monclient: dump: 2023-11-02T08:35:36.947 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:36.947 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:36.948 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.948 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:36.948 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:36.948 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:36.949 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:36.949 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:36.949 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:36.949 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:36.949 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be8cc4640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:36.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be8cc4640 10 monclient: _renew_subs 2023-11-02T08:35:36.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be8cc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:36.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be8cc4640 1 -- 172.21.15.178:0/3723254508 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be41438a0 con 0x7f2be406fe90 2023-11-02T08:35:36.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.937+0000 7f2be8cc4640 1 librados: init done 2023-11-02T08:35:36.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.938+0000 7f2be8cc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:36.951 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.938+0000 7f2be8cc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:36.951 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.938+0000 7f2be8cc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:36.951 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:36.938+0000 7f2be8cc4640 1 -- 172.21.15.178:0/3723254508 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be404e040 con 0x7f2be406fe90 2023-11-02T08:35:37.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.087+0000 7f2bd2ffd640 1 -- 172.21.15.178:0/3723254508 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 2) v1 ==== 44255+0+0 (secure 0 0 0) 0x7f2bd404d3f0 con 0x7f2be406fe90 2023-11-02T08:35:37.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.097+0000 7f2bd2ffd640 1 -- 172.21.15.178:0/3723254508 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f2bd4030330 con 0x7f2be406fe90 2023-11-02T08:35:37.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.097+0000 7f2bd2ffd640 10 client.4156.objecter ms_dispatch 0x7f2be4000b90 osd_map(8..8 src has 1..8) v4 2023-11-02T08:35:37.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.097+0000 7f2bd2ffd640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2023-11-02T08:35:37.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.097+0000 7f2bd2ffd640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2023-11-02T08:35:37.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.097+0000 7f2bd2ffd640 20 client.4156.objecter dump_active .. 0 homeless 2023-11-02T08:35:37.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.097+0000 7f2bd2ffd640 1 -- 172.21.15.178:0/3723254508 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f2bd404c350 con 0x7f2be406fe90 2023-11-02T08:35:37.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.097+0000 7f2bd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:37.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.097+0000 7f2bd2ffd640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:37.181 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.177+0000 7f2be8cc4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bfab5b18-727d-4c9f-a9f0-30d624e82282", "id": 6}] 2023-11-02T08:35:37.181 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.177+0000 7f2be8cc4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bfab5b18-727d-4c9f-a9f0-30d624e82282", "id": 6}] 2023-11-02T08:35:37.181 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.177+0000 7f2be8cc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:37.181 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.177+0000 7f2be8cc4640 1 -- 172.21.15.178:0/3723254508 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bfab5b18-727d-4c9f-a9f0-30d624e82282", "id": 6} v 0) v1 -- 0x7f2be413cf20 con 0x7f2be406fe90 2023-11-02T08:35:37.368 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.364+0000 7f2bd2ffd640 1 -- 172.21.15.178:0/3723254508 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfab5b18-727d-4c9f-a9f0-30d624e82282", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2be413cf20 con 0x7f2be406fe90 2023-11-02T08:35:37.368 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.364+0000 7f2bd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bfab5b18-727d-4c9f-a9f0-30d624e82282", "id": 6}] 2023-11-02T08:35:37.368 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.364+0000 7f2bd2ffd640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:37.369 INFO:teuthology.orchestra.run.smithi178.stdout:6 2023-11-02T08:35:37.370 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 10 librados: watch_flush enter 2023-11-02T08:35:37.370 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 10 librados: watch_flush exit 2023-11-02T08:35:37.371 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 20 client.4156.objecter shutdown clearing up homeless session... 2023-11-02T08:35:37.371 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 10 client.4156.objecter successfully canceled tick 2023-11-02T08:35:37.371 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 10 monclient: shutdown 2023-11-02T08:35:37.371 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:37.371 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 1 -- 172.21.15.178:0/3723254508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 msgr2=0x7f2be41356c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:37.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 1 --2- 172.21.15.178:0/3723254508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 0x7f2be41356c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2bd40090b0 tx=0x7f2bd40046c0 comp rx=0 tx=0).stop 2023-11-02T08:35:37.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 1 -- 172.21.15.178:0/3723254508 shutdown_connections 2023-11-02T08:35:37.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 1 --2- 172.21.15.178:0/3723254508 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2be413f5f0 0x7f2be4141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:37.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 1 --2- 172.21.15.178:0/3723254508 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2be413c3d0 0x7f2be4137020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:37.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 1 --2- 172.21.15.178:0/3723254508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2be406fe90 0x7f2be41356c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:37.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 1 -- 172.21.15.178:0/3723254508 >> 172.21.15.178:0/3723254508 conn(0x7f2be40b54e0 msgr2=0x7f2be40b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:37.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 1 -- 172.21.15.178:0/3723254508 shutdown_connections 2023-11-02T08:35:37.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.365+0000 7f2be8cc4640 1 -- 172.21.15.178:0/3723254508 wait complete. 2023-11-02T08:35:37.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.366+0000 7f2be8cc4640 1 librados: shutdown 2023-11-02T08:35:37.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new dedf0003-4a0c-4672-81f3-3808816fca03 7 2023-11-02T08:35:37.479 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.475+0000 7f8bf9e51640 10 monclient: get_monmap_and_config 2023-11-02T08:35:37.479 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.475+0000 7f8bf9e51640 10 monclient: build_initial_monmap 2023-11-02T08:35:37.479 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.475+0000 7f8bf9e51640 10 monclient: monmap: 2023-11-02T08:35:37.480 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:37.480 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:37.480 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:35:37.476893+0000 2023-11-02T08:35:37.480 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:35:37.476893+0000 2023-11-02T08:35:37.480 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:37.481 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:37.481 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:37.481 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:37.481 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:37.481 INFO:teuthology.orchestra.run.smithi178.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:37.482 INFO:teuthology.orchestra.run.smithi178.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:37.482 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:37.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.476+0000 7f8bf9e51640 1 Processor -- start 2023-11-02T08:35:37.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.476+0000 7f8bf9e51640 1 -- start start 2023-11-02T08:35:37.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.476+0000 7f8bf9e51640 10 monclient: init 2023-11-02T08:35:37.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:37.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-11-02T08:35:37.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8bf40b4570 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:37.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b0ed0 0x7f8bf4144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:37.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 10 monclient(hunting): picked mon.noname-b con 0x7f8bf40b0ed0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:37.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bf413e520 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:37.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf40a9780 con 0x7f8bf413e520 2023-11-02T08:35:37.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf40aa1d0 con 0x7f8bf40b4570 2023-11-02T08:35:37.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:37.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40b4950 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:37.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf9e51640 10 monclient(hunting): authenticate will time out at 1289.804363s 2023-11-02T08:35:37.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf37fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f8bf40b4570 0x7f8bf40a9070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:40050/0 (socket says 172.21.15.178:40050) 2023-11-02T08:35:37.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.477+0000 7f8bf37fe640 1 -- 172.21.15.178:0/2526991769 learned_addr learned my addr 172.21.15.178:0/2526991769 (peer_addr_for_me v1:172.21.15.178:0/0) 2023-11-02T08:35:37.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.478+0000 7f8bf2ffd640 1 --2- 172.21.15.178:0/2526991769 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b0ed0 0x7f8bf4144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:37.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.478+0000 7f8bf2ffd640 10 monclient(hunting): get_auth_request con 0x7f8bf40b0ed0 auth_method 0 2023-11-02T08:35:37.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.478+0000 7f8bf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:37.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.478+0000 7f8bf2ffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:37.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.478+0000 7f8bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:37.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.478+0000 7f8bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:37.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.478+0000 7f8bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:37.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.478+0000 7f8bf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:37.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf27fc640 1 -- 172.21.15.178:0/2526991769 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506889338 0 0) 0x7f8bf40aa1d0 con 0x7f8bf40b4570 2023-11-02T08:35:37.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf27fc640 10 monclient(hunting): _negotiate 2023-11-02T08:35:37.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf27fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:37.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf27fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:37.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf27fc640 10 monclient(hunting): my global_id is 4171 2023-11-02T08:35:37.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf27fc640 1 -- 172.21.15.178:0/2526991769 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bdc003520 con 0x7f8bf40b4570 2023-11-02T08:35:37.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf2ffd640 10 monclient(hunting): handle_auth_done global_id 4142 payload 274 2023-11-02T08:35:37.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf2ffd640 1 -- 172.21.15.178:0/2526991769 >> v1:172.21.15.178:6789/0 conn(0x7f8bf40b4570 legacy=0x7f8bf40a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:37.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf2ffd640 1 -- 172.21.15.178:0/2526991769 >> v1:172.21.15.151:6789/0 conn(0x7f8bf413e520 legacy=0x7f8bf413e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:37.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf2ffd640 10 monclient: _finish_hunting 0 2023-11-02T08:35:37.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf2ffd640 1 monclient: found mon.noname-b 2023-11-02T08:35:37.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:37.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:35:37.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.480+0000 7f8bf2ffd640 1 -- 172.21.15.178:0/2526991769 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40aa4b0 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.482+0000 7f8bf2ffd640 1 --2- 172.21.15.178:0/2526991769 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b0ed0 0x7f8bf4144d40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8be80097c0 tx=0x7f8be802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dd34a419e1451dc server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:37.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 1 -- 172.21.15.178:0/2526991769 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be80316d0 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:37.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:35:37.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 10 monclient: dump: 2023-11-02T08:35:37.496 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:37.496 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:37.496 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:37.497 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:37.497 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:37.497 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:37.497 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:37.497 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:37.498 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:37.498 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:37.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 10 monclient: _finish_auth 0 2023-11-02T08:35:37.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 10 monclient: _check_auth_tickets 2023-11-02T08:35:37.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:37.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 1 -- 172.21.15.178:0/2526991769 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be8031830 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 5 monclient: authenticate success, global_id 4142 2023-11-02T08:35:37.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:35:37.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:37.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:37.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 1 -- 172.21.15.178:0/2526991769 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be8031bc0 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:37.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 4 set_mon_vals no callback set 2023-11-02T08:35:37.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:35:37.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf27fc640 10 monclient: dump: 2023-11-02T08:35:37.501 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:37.501 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:37.501 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:37.501 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:37.501 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:37.502 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:37.502 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:37.502 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:37.502 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:37.502 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:37.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 10 monclient: shutdown 2023-11-02T08:35:37.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:37.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 1 -- 172.21.15.178:0/2526991769 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b0ed0 msgr2=0x7f8bf4144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:37.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 1 --2- 172.21.15.178:0/2526991769 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b0ed0 0x7f8bf4144d40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8be80097c0 tx=0x7f8be802b770 comp rx=0 tx=0).stop 2023-11-02T08:35:37.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 1 -- 172.21.15.178:0/2526991769 shutdown_connections 2023-11-02T08:35:37.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 1 --2- 172.21.15.178:0/2526991769 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b0ed0 0x7f8bf4144d40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:37.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.486+0000 7f8bf9e51640 1 -- 172.21.15.178:0/2526991769 >> 172.21.15.178:0/2526991769 conn(0x7f8bf4065d50 msgr2=0x7f8bf4066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:37.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.487+0000 7f8bf9e51640 1 -- 172.21.15.178:0/2526991769 shutdown_connections 2023-11-02T08:35:37.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.487+0000 7f8bf9e51640 1 -- 172.21.15.178:0/2526991769 wait complete. 2023-11-02T08:35:37.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.487+0000 7f8bf9e51640 10 monclient: build_initial_monmap 2023-11-02T08:35:37.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.487+0000 7f8bf9e51640 10 monclient: monmap: 2023-11-02T08:35:37.505 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2023-11-02T08:35:37.505 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:37.505 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2023-11-02T08:35:37.505 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2023-11-02T08:35:37.505 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:37.506 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:37.506 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:37.506 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:37.506 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:37.506 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:37.507 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.487+0000 7f8bf9e51640 1 librados: starting msgr at 2023-11-02T08:35:37.507 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.487+0000 7f8bf9e51640 1 librados: starting objecter 2023-11-02T08:35:37.507 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.487+0000 7f8bf9e51640 1 Processor -- start 2023-11-02T08:35:37.507 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 1 -- start start 2023-11-02T08:35:37.508 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 1 librados: setting wanted keys 2023-11-02T08:35:37.508 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 1 librados: calling monclient init 2023-11-02T08:35:37.508 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient: init 2023-11-02T08:35:37.508 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:37.508 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-02T08:35:37.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bf40b0ed0 0x7f8bf40ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:37.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient(hunting): picked mon.noname-b con 0x7f8bf40b0ed0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:37.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b4570 0x7f8bf40ab320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:37.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient(hunting): picked mon.noname-c con 0x7f8bf40b4570 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:37.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bf413e520 0x7f8bf41d6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:37.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient(hunting): picked mon.noname-a con 0x7f8bf413e520 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:37.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:37.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40aaf00 con 0x7f8bf413e520 2023-11-02T08:35:37.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:37.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40ac6b0 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.511 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:37.511 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf41d6750 con 0x7f8bf40b4570 2023-11-02T08:35:37.511 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:37.511 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.488+0000 7f8bf9e51640 10 monclient(hunting): authenticate will time out at 1289.815803s 2023-11-02T08:35:37.511 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf37fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bf40b0ed0 0x7f8bf40ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:37.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf2ffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b4570 0x7f8bf40ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:37.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf37fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bf40b0ed0 0x7f8bf40ac170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:42740/0 (socket says 172.21.15.178:42740) 2023-11-02T08:35:37.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf37fe640 1 -- 172.21.15.178:0/3580934751 learned_addr learned my addr 172.21.15.178:0/3580934751 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-02T08:35:37.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf37fe640 10 monclient(hunting): get_auth_request con 0x7f8bf40b0ed0 auth_method 0 2023-11-02T08:35:37.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:37.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf37fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:37.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf37fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:37.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf2ffd640 10 monclient(hunting): get_auth_request con 0x7f8bf40b4570 auth_method 0 2023-11-02T08:35:37.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:37.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf2ffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:37.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.489+0000 7f8bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:37.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.490+0000 7f8bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:37.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.490+0000 7f8bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:37.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.490+0000 7f8bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:37.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.490+0000 7f8bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:37.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.490+0000 7f8bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:37.515 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.490+0000 7f8bf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:37.516 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 10 monclient(hunting): handle_auth_done global_id 4174 payload 824 2023-11-02T08:35:37.517 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 1 -- 172.21.15.178:0/3580934751 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b4570 msgr2=0x7f8bf40ab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:37.517 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b4570 0x7f8bf40ab320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:37.517 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 1 -- 172.21.15.178:0/3580934751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bf413e520 msgr2=0x7f8bf41d6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:37.517 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bf413e520 0x7f8bf41d6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:37.518 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 10 monclient: _finish_hunting 0 2023-11-02T08:35:37.518 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 1 monclient: found mon.noname-b 2023-11-02T08:35:37.518 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:37.518 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:35:37.518 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.492+0000 7f8bf37fe640 1 -- 172.21.15.178:0/3580934751 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be80093a0 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.494+0000 7f8bf37fe640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bf40b0ed0 0x7f8bf40ac170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8be400c500 tx=0x7f8be400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:37.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.494+0000 7f8bdbfff640 10 client.?.objecter ms_handle_connect 0x7f8bf40b0ed0 2023-11-02T08:35:37.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.494+0000 7f8bdbfff640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:37.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 1 -- 172.21.15.178:0/3580934751 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4005ce0 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:37.520 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:35:37.520 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 10 monclient: dump: 2023-11-02T08:35:37.520 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:37.520 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:37.520 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:37.521 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:37.521 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:37.521 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:37.521 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:37.521 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:37.521 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:37.522 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:37.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 10 monclient: _finish_auth 0 2023-11-02T08:35:37.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 10 monclient: _check_auth_tickets 2023-11-02T08:35:37.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:37.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 1 -- 172.21.15.178:0/3580934751 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be4005e40 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf9e51640 5 monclient: authenticate success, global_id 4174 2023-11-02T08:35:37.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf9e51640 10 monclient: _renew_subs 2023-11-02T08:35:37.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf9e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:37.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf4140790 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:37.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 1 -- 172.21.15.178:0/3580934751 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4007330 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:37.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:37.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bdbfff640 10 monclient: dump: 2023-11-02T08:35:37.524 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2023-11-02T08:35:37.525 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:37.525 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:37.525 INFO:teuthology.orchestra.run.smithi178.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:37.525 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:37.525 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2023-11-02T08:35:37.526 INFO:teuthology.orchestra.run.smithi178.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:37.526 INFO:teuthology.orchestra.run.smithi178.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:37.526 INFO:teuthology.orchestra.run.smithi178.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:37.526 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-02T08:35:37.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf9e51640 10 client.4174.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:37.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf9e51640 10 monclient: _renew_subs 2023-11-02T08:35:37.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf9e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:37.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf40aba40 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf9e51640 1 librados: init done 2023-11-02T08:35:37.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.498+0000 7f8bf1ffb640 4 set_mon_vals no callback set 2023-11-02T08:35:37.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.499+0000 7f8bf9e51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:37.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.499+0000 7f8bf9e51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:37.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.499+0000 7f8bf9e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:37.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.499+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4067920 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.599+0000 7f8bdbfff640 1 -- 172.21.15.178:0/3580934751 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 2) v1 ==== 44255+0+0 (secure 0 0 0) 0x7f8be4007490 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.609 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.605+0000 7f8bdbfff640 1 -- 172.21.15.178:0/3580934751 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f8be4007740 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.609 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.605+0000 7f8bdbfff640 10 client.4174.objecter ms_dispatch 0x7f8bf4000b90 osd_map(9..9 src has 1..9) v4 2023-11-02T08:35:37.609 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.605+0000 7f8bdbfff640 3 client.4174.objecter handle_osd_map got epochs [9,9] > 0 2023-11-02T08:35:37.610 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.605+0000 7f8bdbfff640 3 client.4174.objecter handle_osd_map decoding full epoch 9 2023-11-02T08:35:37.610 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.605+0000 7f8bdbfff640 20 client.4174.objecter dump_active .. 0 homeless 2023-11-02T08:35:37.610 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.605+0000 7f8bdbfff640 1 -- 172.21.15.178:0/3580934751 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f8be4019910 con 0x7f8bf40b0ed0 2023-11-02T08:35:37.610 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.606+0000 7f8bdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:37.610 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.606+0000 7f8bdbfff640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:37.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.676+0000 7f8bf9e51640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dedf0003-4a0c-4672-81f3-3808816fca03", "id": 7}] 2023-11-02T08:35:37.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.676+0000 7f8bf9e51640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dedf0003-4a0c-4672-81f3-3808816fca03", "id": 7}] 2023-11-02T08:35:37.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.676+0000 7f8bf9e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:37.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:37.676+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dedf0003-4a0c-4672-81f3-3808816fca03", "id": 7} v 0) v1 -- 0x7f8bf40b53d0 con 0x7f8bf40b0ed0 2023-11-02T08:35:38.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.137+0000 7f8bdbfff640 1 -- 172.21.15.178:0/3580934751 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mgrmap(e 3) v1 ==== 88445+0+0 (secure 0 0 0) 0x7f8be401b8d0 con 0x7f8bf40b0ed0 2023-11-02T08:35:38.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.138+0000 7f8bdbfff640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8be006c0f0 0x7f8be006e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:38.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.138+0000 7f8bf2ffd640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8be006c0f0 0x7f8be006e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:38.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.138+0000 7f8bf2ffd640 10 monclient: get_auth_request con 0x7f8be006c0f0 auth_method 0 2023-11-02T08:35:38.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.138+0000 7f8bdbfff640 10 client.4174.objecter ms_handle_connect 0x7f8be006c0f0 2023-11-02T08:35:38.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.138+0000 7f8bf2ffd640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8be006c0f0 0x7f8be006e4e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8be8036000 tx=0x7f8be8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:38.171 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.168+0000 7f8bdbfff640 1 -- 172.21.15.178:0/3580934751 <== mon.1 v2:172.21.15.178:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dedf0003-4a0c-4672-81f3-3808816fca03", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8be4055de0 con 0x7f8bf40b0ed0 2023-11-02T08:35:38.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.168+0000 7f8bdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dedf0003-4a0c-4672-81f3-3808816fca03", "id": 7}] 2023-11-02T08:35:38.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.168+0000 7f8bdbfff640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:38.172 INFO:teuthology.orchestra.run.smithi178.stdout:7 2023-11-02T08:35:38.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 10 librados: watch_flush enter 2023-11-02T08:35:38.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 10 librados: watch_flush exit 2023-11-02T08:35:38.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 20 client.4174.objecter shutdown clearing up homeless session... 2023-11-02T08:35:38.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 10 client.4174.objecter successfully canceled tick 2023-11-02T08:35:38.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8be006c0f0 msgr2=0x7f8be006e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:38.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8be006c0f0 0x7f8be006e4e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8be8036000 tx=0x7f8be8008030 comp rx=0 tx=0).stop 2023-11-02T08:35:38.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 10 monclient: shutdown 2023-11-02T08:35:38.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:38.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bf40b0ed0 msgr2=0x7f8bf40ac170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:38.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bf40b0ed0 0x7f8bf40ac170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8be400c500 tx=0x7f8be400c9c0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 shutdown_connections 2023-11-02T08:35:38.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8bf413e520 0x7f8bf41d6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8bf40b4570 0x7f8bf40ab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8be006c0f0 0x7f8be006e4e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 --2- 172.21.15.178:0/3580934751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bf40b0ed0 0x7f8bf40ac170 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 >> 172.21.15.178:0/3580934751 conn(0x7f8bf4065d50 msgr2=0x7f8bf40b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:38.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 shutdown_connections 2023-11-02T08:35:38.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 -- 172.21.15.178:0/3580934751 wait complete. 2023-11-02T08:35:38.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-02T08:35:38.169+0000 7f8bf9e51640 1 librados: shutdown 2023-11-02T08:35:38.185 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-02T08:35:38.185 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-02T08:35:38.186 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-11-02T08:35:38.188 INFO:tasks.ceph.osd.0:Started 2023-11-02T08:35:38.189 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-02T08:35:38.189 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-02T08:35:38.189 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-11-02T08:35:38.192 INFO:tasks.ceph.osd.1:Started 2023-11-02T08:35:38.192 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-02T08:35:38.193 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-02T08:35:38.193 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-11-02T08:35:38.195 INFO:tasks.ceph.osd.2:Started 2023-11-02T08:35:38.195 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-02T08:35:38.196 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-02T08:35:38.196 DEBUG:teuthology.orchestra.run.smithi151:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-11-02T08:35:38.198 INFO:tasks.ceph.osd.3:Started 2023-11-02T08:35:38.199 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-02T08:35:38.199 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-02T08:35:38.199 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-11-02T08:35:38.202 INFO:tasks.ceph.osd.4:Started 2023-11-02T08:35:38.202 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-02T08:35:38.202 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-02T08:35:38.203 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-11-02T08:35:38.205 INFO:tasks.ceph.osd.5:Started 2023-11-02T08:35:38.205 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-02T08:35:38.205 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-02T08:35:38.206 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-11-02T08:35:38.208 INFO:tasks.ceph.osd.6:Started 2023-11-02T08:35:38.208 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-11-02T08:35:38.209 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-02T08:35:38.209 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-11-02T08:35:38.211 INFO:tasks.ceph.osd.7:Started 2023-11-02T08:35:38.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-02T08:35:38.306 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:38.303+0000 7fb19d883640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-02T08:35:38.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.358+0000 7f98105d3640 10 monclient: get_monmap_and_config 2023-11-02T08:35:38.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.358+0000 7f98105d3640 10 monclient: build_initial_monmap 2023-11-02T08:35:38.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.358+0000 7f98105d3640 10 monclient: monmap: 2023-11-02T08:35:38.362 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:35:38.363 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:38.363 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:35:38.359136+0000 2023-11-02T08:35:38.363 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:35:38.359136+0000 2023-11-02T08:35:38.363 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:38.363 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:38.363 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:38.364 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:38.364 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:38.364 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:38.364 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:38.364 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:38.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.358+0000 7f98105d3640 1 Processor -- start 2023-11-02T08:35:38.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.358+0000 7f98105d3640 1 -- start start 2023-11-02T08:35:38.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.358+0000 7f98105d3640 10 monclient: init 2023-11-02T08:35:38.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:38.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-11-02T08:35:38.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98080b4260 0x7f98080a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:38.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 10 monclient(hunting): picked mon.noname-b con 0x7f98080b4260 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:38.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9808141730 0x7f9808141b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:38.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 10 monclient(hunting): picked mon.noname-a con 0x7f9808141730 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:38.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98080b1a00 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:38.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98080a9210 con 0x7f98080b1a00 2023-11-02T08:35:38.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:38.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98080aa550 con 0x7f9808141730 2023-11-02T08:35:38.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:38.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98080b4640 con 0x7f98080b4260 2023-11-02T08:35:38.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:38.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.359+0000 7f98105d3640 10 monclient(hunting): authenticate will time out at 1309.715598s 2023-11-02T08:35:38.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.364+0000 7f980db6e640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9808141730 0x7f9808141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:38.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.364+0000 7f980db6e640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9808141730 0x7f9808141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36638/0 (socket says 172.21.15.151:36638) 2023-11-02T08:35:38.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.364+0000 7f980db6e640 1 -- 172.21.15.151:0/776770771 learned_addr learned my addr 172.21.15.151:0/776770771 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:35:38.372 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.364+0000 7f980db6e640 10 monclient(hunting): get_auth_request con 0x7f9808141730 auth_method 0 2023-11-02T08:35:38.372 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.364+0000 7f980db6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:38.372 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.364+0000 7f980db6e640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:38.372 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.364+0000 7f980db6e640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:38.372 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.365+0000 7f980db6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:38.372 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.365+0000 7f980db6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.365+0000 7f980db6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.367+0000 7f980db6e640 10 monclient(hunting): handle_auth_done global_id 4143 payload 274 2023-11-02T08:35:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.367+0000 7f980db6e640 1 -- 172.21.15.151:0/776770771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98080b4260 msgr2=0x7f98080a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.367+0000 7f980db6e640 1 --2- 172.21.15.151:0/776770771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98080b4260 0x7f98080a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.367+0000 7f980db6e640 1 -- 172.21.15.151:0/776770771 >> v1:172.21.15.178:6789/0 conn(0x7f98080b1a00 legacy=0x7f98080b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:38.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.367+0000 7f980db6e640 10 monclient: _finish_hunting 0 2023-11-02T08:35:38.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.367+0000 7f980db6e640 1 monclient: found mon.noname-a 2023-11-02T08:35:38.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.367+0000 7f980db6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:38.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.367+0000 7f980db6e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:38.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.367+0000 7f980db6e640 1 -- 172.21.15.151:0/776770771 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98080aa750 con 0x7f9808141730 2023-11-02T08:35:38.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.369+0000 7f980db6e640 1 --2- 172.21.15.151:0/776770771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9808141730 0x7f9808141b70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97f80098a0 tx=0x7f97f802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a499151ded67a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 1 -- 172.21.15.151:0/776770771 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f80317c0 con 0x7f9808141730 2023-11-02T08:35:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:38.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:38.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 10 monclient: dump: 2023-11-02T08:35:38.380 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:35:38.380 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:38.381 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:38.381 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:38.381 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:38.382 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:38.382 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:38.382 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:38.383 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:38.383 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:38.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 10 monclient: _finish_auth 0 2023-11-02T08:35:38.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 10 monclient: _check_auth_tickets 2023-11-02T08:35:38.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:38.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 1 -- 172.21.15.151:0/776770771 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f8031920 con 0x7f9808141730 2023-11-02T08:35:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 1 -- 172.21.15.151:0/776770771 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8031cb0 con 0x7f9808141730 2023-11-02T08:35:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:38.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f980d36d640 10 monclient: dump: 2023-11-02T08:35:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:35:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:38.386 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:38.387 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:38.387 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:38.387 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:38.387 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:38.387 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f98105d3640 5 monclient: authenticate success, global_id 4143 2023-11-02T08:35:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f98105d3640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.373+0000 7f98105d3640 4 set_mon_vals no callback set 2023-11-02T08:35:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 10 monclient: shutdown 2023-11-02T08:35:38.390 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:38.390 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 1 -- 172.21.15.151:0/776770771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9808141730 msgr2=0x7f9808141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:38.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 1 --2- 172.21.15.151:0/776770771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9808141730 0x7f9808141b70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97f80098a0 tx=0x7f97f802b860 comp rx=0 tx=0).stop 2023-11-02T08:35:38.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 1 -- 172.21.15.151:0/776770771 shutdown_connections 2023-11-02T08:35:38.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 1 --2- 172.21.15.151:0/776770771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9808141730 0x7f9808141b70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 1 --2- 172.21.15.151:0/776770771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98080b4260 0x7f98080a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 1 -- 172.21.15.151:0/776770771 >> 172.21.15.151:0/776770771 conn(0x7f9808065d50 msgr2=0x7f9808066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:38.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 1 -- 172.21.15.151:0/776770771 shutdown_connections 2023-11-02T08:35:38.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.374+0000 7f98105d3640 1 -- 172.21.15.151:0/776770771 wait complete. 2023-11-02T08:35:38.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.375+0000 7f98105d3640 10 monclient: build_initial_monmap 2023-11-02T08:35:38.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.375+0000 7f98105d3640 10 monclient: monmap: 2023-11-02T08:35:38.392 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:35:38.393 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:38.393 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:35:38.393 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:35:38.393 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:38.393 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:38.394 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:38.394 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:38.394 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:38.394 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:38.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.376+0000 7f98105d3640 1 librados: starting msgr at 2023-11-02T08:35:38.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.376+0000 7f98105d3640 1 librados: starting objecter 2023-11-02T08:35:38.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.376+0000 7f98105d3640 1 Processor -- start 2023-11-02T08:35:38.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.376+0000 7f98105d3640 1 -- start start 2023-11-02T08:35:38.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.376+0000 7f98105d3640 1 librados: setting wanted keys 2023-11-02T08:35:38.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.376+0000 7f98105d3640 1 librados: calling monclient init 2023-11-02T08:35:38.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.376+0000 7f98105d3640 10 monclient: init 2023-11-02T08:35:38.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:38.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:35:38.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98080b1a00 0x7f98081db140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:38.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient(hunting): picked mon.noname-b con 0x7f98080b1a00 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:38.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98080b4260 0x7f98081db680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:38.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient(hunting): picked mon.noname-a con 0x7f98080b4260 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:38.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9808141730 0x7f98081de2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:38.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient(hunting): picked mon.noname-c con 0x7f9808141730 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:38.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:38.401 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980813dc50 con 0x7f98080b4260 2023-11-02T08:35:38.401 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:38.401 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98081de7e0 con 0x7f98080b1a00 2023-11-02T08:35:38.401 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:38.401 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98081de950 con 0x7f9808141730 2023-11-02T08:35:38.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:38.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.377+0000 7f98105d3640 10 monclient(hunting): authenticate will time out at 1309.733831s 2023-11-02T08:35:38.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.380+0000 7f980e36f640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98080b1a00 0x7f98081db140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:38.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.384+0000 7f980e36f640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98080b1a00 0x7f98081db140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:47962/0 (socket says 172.21.15.151:47962) 2023-11-02T08:35:38.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.384+0000 7f980e36f640 1 -- 172.21.15.151:0/3014856853 learned_addr learned my addr 172.21.15.151:0/3014856853 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:35:38.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.384+0000 7f980e36f640 10 monclient(hunting): get_auth_request con 0x7f98080b1a00 auth_method 0 2023-11-02T08:35:38.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.384+0000 7f980e36f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:38.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.384+0000 7f980e36f640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:38.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.384+0000 7f980e36f640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:38.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.385+0000 7f980e36f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:38.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.385+0000 7f980e36f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:38.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.385+0000 7f980e36f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:38.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.387+0000 7f980eb70640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9808141730 0x7f98081de2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:38.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.387+0000 7f980eb70640 10 monclient(hunting): get_auth_request con 0x7f9808141730 auth_method 0 2023-11-02T08:35:38.405 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.387+0000 7f980eb70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:38.405 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.387+0000 7f980eb70640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:38.405 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.387+0000 7f980eb70640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:38.405 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.388+0000 7f980eb70640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:38.405 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.388+0000 7f980eb70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:38.406 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.388+0000 7f980eb70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:38.406 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.388+0000 7f980db6e640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98080b4260 0x7f98081db680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:38.406 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.389+0000 7f980db6e640 10 monclient(hunting): get_auth_request con 0x7f98080b4260 auth_method 0 2023-11-02T08:35:38.406 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.389+0000 7f980db6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:38.406 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.389+0000 7f980db6e640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.389+0000 7f980db6e640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.397+0000 7f980db6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.397+0000 7f980db6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:38.408 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.397+0000 7f980db6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:38.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 10 monclient(hunting): handle_auth_done global_id 4177 payload 824 2023-11-02T08:35:38.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 1 -- 172.21.15.151:0/3014856853 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9808141730 msgr2=0x7f98081de2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:38.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9808141730 0x7f98081de2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 1 -- 172.21.15.151:0/3014856853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98080b4260 msgr2=0x7f98081db680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:38.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98080b4260 0x7f98081db680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:38.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 10 monclient: _finish_hunting 0 2023-11-02T08:35:38.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 1 monclient: found mon.noname-b 2023-11-02T08:35:38.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:38.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:35:38.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.410+0000 7f980e36f640 1 -- 172.21.15.151:0/3014856853 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f80094d0 con 0x7f98080b1a00 2023-11-02T08:35:38.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.430+0000 7f980e36f640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98080b1a00 0x7f98081db140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f980400b4a0 tx=0x7f980400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:38.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.430+0000 7f97feffd640 10 client.?.objecter ms_handle_connect 0x7f98080b1a00 2023-11-02T08:35:38.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.430+0000 7f97feffd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:38.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.446+0000 7f97feffd640 1 -- 172.21.15.151:0/3014856853 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9804010610 con 0x7f98080b1a00 2023-11-02T08:35:38.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.446+0000 7f97feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:38.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.446+0000 7f97feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:35:38.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.446+0000 7f97feffd640 10 monclient: dump: 2023-11-02T08:35:38.451 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:35:38.451 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:38.451 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:38.451 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:38.452 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:38.452 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:38.452 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:38.452 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:38.452 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:38.452 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:38.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.446+0000 7f97feffd640 10 monclient: _finish_auth 0 2023-11-02T08:35:38.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.446+0000 7f97feffd640 10 monclient: _check_auth_tickets 2023-11-02T08:35:38.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.446+0000 7f97feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:38.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.446+0000 7f97feffd640 1 -- 172.21.15.151:0/3014856853 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9804010770 con 0x7f98080b1a00 2023-11-02T08:35:38.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.446+0000 7f97feffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:38.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f97feffd640 1 -- 172.21.15.151:0/3014856853 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9804010b20 con 0x7f98080b1a00 2023-11-02T08:35:38.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f97feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:38.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f97feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:38.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f97feffd640 10 monclient: dump: 2023-11-02T08:35:38.454 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:35:38.454 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:38.455 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:38.455 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:38.455 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:38.455 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:38.455 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:38.456 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:38.456 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:38.456 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:35:38.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f980cb6c640 4 set_mon_vals no callback set 2023-11-02T08:35:38.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f98105d3640 5 monclient: authenticate success, global_id 4177 2023-11-02T08:35:38.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f98105d3640 10 monclient: _renew_subs 2023-11-02T08:35:38.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f98105d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:38.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98081debd0 con 0x7f98080b1a00 2023-11-02T08:35:38.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f98105d3640 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:38.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f98105d3640 10 monclient: _renew_subs 2023-11-02T08:35:38.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f98105d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:38.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980806d880 con 0x7f98080b1a00 2023-11-02T08:35:38.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.447+0000 7f98105d3640 1 librados: init done 2023-11-02T08:35:38.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.448+0000 7f98105d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:38.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.448+0000 7f98105d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:38.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.448+0000 7f98105d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:38.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.449+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98081dadd0 con 0x7f98080b1a00 2023-11-02T08:35:38.523 INFO:tasks.ceph.mgr.x.smithi178.stderr:2023-11-02T08:35:38.520+0000 7fb19d883640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-02T08:35:38.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.879+0000 7f97feffd640 1 -- 172.21.15.151:0/3014856853 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 3) v1 ==== 88445+0+0 (secure 0 0 0) 0x7f9804010c80 con 0x7f98080b1a00 2023-11-02T08:35:38.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.879+0000 7f97feffd640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f97ec06bc60 0x7f97ec06e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:38.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.879+0000 7f980db6e640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f97ec06bc60 0x7f97ec06e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:38.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.879+0000 7f980db6e640 10 monclient: get_auth_request con 0x7f97ec06bc60 auth_method 0 2023-11-02T08:35:38.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.880+0000 7f980db6e640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f97ec06bc60 0x7f97ec06e120 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f97f8036000 tx=0x7f97f803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:38.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.880+0000 7f97feffd640 10 client.4177.objecter ms_handle_connect 0x7f97ec06bc60 2023-11-02T08:35:38.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.900+0000 7f97feffd640 1 -- 172.21.15.151:0/3014856853 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9804057c30 con 0x7f98080b1a00 2023-11-02T08:35:38.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.900+0000 7f97feffd640 10 client.4177.objecter ms_dispatch 0x7f9808000b90 osd_map(10..10 src has 1..10) v4 2023-11-02T08:35:38.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.900+0000 7f97feffd640 3 client.4177.objecter handle_osd_map got epochs [10,10] > 0 2023-11-02T08:35:38.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.900+0000 7f97feffd640 3 client.4177.objecter handle_osd_map decoding full epoch 10 2023-11-02T08:35:38.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.900+0000 7f97feffd640 20 client.4177.objecter dump_active .. 0 homeless 2023-11-02T08:35:38.929 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.925+0000 7f97feffd640 1 -- 172.21.15.151:0/3014856853 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f9804053f70 con 0x7f98080b1a00 2023-11-02T08:35:38.929 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.926+0000 7f97feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:38.929 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:38.926+0000 7f97feffd640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:39.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.276+0000 7f98105d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:39.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.276+0000 7f98105d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:39.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.276+0000 7f98105d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:39.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.276+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9808067920 con 0x7f98080b1a00 2023-11-02T08:35:39.352 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.349+0000 7f97feffd640 1 -- 172.21.15.151:0/3014856853 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f980401e020 con 0x7f98080b1a00 2023-11-02T08:35:39.353 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.349+0000 7f97feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:39.353 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.349+0000 7f97feffd640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:39.353 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:35:39.353 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":10,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:35:37.853543+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:35:39.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.352+0000 7f98105d3640 10 librados: watch_flush enter 2023-11-02T08:35:39.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.352+0000 7f98105d3640 10 librados: watch_flush exit 2023-11-02T08:35:39.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.352+0000 7f98105d3640 20 client.4177.objecter shutdown clearing up homeless session... 2023-11-02T08:35:39.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.352+0000 7f98105d3640 10 client.4177.objecter successfully canceled tick 2023-11-02T08:35:39.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f97ec06bc60 msgr2=0x7f97ec06e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:39.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f97ec06bc60 0x7f97ec06e120 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f97f8036000 tx=0x7f97f803d040 comp rx=0 tx=0).stop 2023-11-02T08:35:39.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 10 monclient: shutdown 2023-11-02T08:35:39.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:39.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98080b1a00 msgr2=0x7f98081db140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:39.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98080b1a00 0x7f98081db140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f980400b4a0 tx=0x7f980400b960 comp rx=0 tx=0).stop 2023-11-02T08:35:39.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 shutdown_connections 2023-11-02T08:35:39.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9808141730 0x7f98081de2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:39.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98080b4260 0x7f98081db680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:39.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f97ec06bc60 0x7f97ec06e120 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:39.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 --2- 172.21.15.151:0/3014856853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98080b1a00 0x7f98081db140 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:39.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 >> 172.21.15.151:0/3014856853 conn(0x7f9808065d50 msgr2=0x7f98080b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:39.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 shutdown_connections 2023-11-02T08:35:39.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 -- 172.21.15.151:0/3014856853 wait complete. 2023-11-02T08:35:39.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:35:39.353+0000 7f98105d3640 1 librados: shutdown 2023-11-02T08:35:39.378 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-02T08:35:39.378 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-02T08:35:40.184 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-02T08:35:40.327 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.324+0000 7fe1365e4640 10 monclient: get_monmap_and_config 2023-11-02T08:35:40.328 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.324+0000 7fe1365e4640 10 monclient: build_initial_monmap 2023-11-02T08:35:40.328 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.324+0000 7fe1365e4640 10 monclient: monmap: 2023-11-02T08:35:40.328 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:35:40.328 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:40.329 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:35:40.325487+0000 2023-11-02T08:35:40.329 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:35:40.325487+0000 2023-11-02T08:35:40.329 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:40.329 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:40.329 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:40.329 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:40.330 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:40.330 INFO:teuthology.misc.health.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:40.330 INFO:teuthology.misc.health.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:40.330 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:40.333 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.325+0000 7fe1365e4640 1 Processor -- start 2023-11-02T08:35:40.333 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.325+0000 7fe1365e4640 1 -- start start 2023-11-02T08:35:40.333 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.325+0000 7fe1365e4640 10 monclient: init 2023-11-02T08:35:40.334 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:40.334 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-11-02T08:35:40.334 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe1300b41c0 0x7fe1300a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:40.334 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe1300b41c0 addr v2:172.21.15.178:3300/0 2023-11-02T08:35:40.334 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe13013e780 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:40.335 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe130141db0 0x7fe13013ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:40.335 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe130141db0 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:40.335 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1300aa360 con 0x7fe13013e780 2023-11-02T08:35:40.335 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:40.335 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1300aa160 con 0x7fe130141db0 2023-11-02T08:35:40.336 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:40.336 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1300b45a0 con 0x7fe1300b41c0 2023-11-02T08:35:40.336 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:40.336 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.326+0000 7fe1365e4640 10 monclient(hunting): authenticate will time out at 1311.682370s 2023-11-02T08:35:40.336 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe134b81640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe130141db0 0x7fe13013ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:40.337 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe12ffff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe1300b41c0 0x7fe1300a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:40.337 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe134b81640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe130141db0 0x7fe13013ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36656/0 (socket says 172.21.15.151:36656) 2023-11-02T08:35:40.337 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe134b81640 1 -- 172.21.15.151:0/3681626139 learned_addr learned my addr 172.21.15.151:0/3681626139 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:35:40.337 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe134b81640 10 monclient(hunting): get_auth_request con 0x7fe130141db0 auth_method 0 2023-11-02T08:35:40.337 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe134b81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:40.338 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe134b81640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:40.338 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe134b81640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:40.338 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe12ffff640 10 monclient(hunting): get_auth_request con 0x7fe1300b41c0 auth_method 0 2023-11-02T08:35:40.338 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe12ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:40.338 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe12ffff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:40.339 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.327+0000 7fe12ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:40.339 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.328+0000 7fe134b81640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:40.339 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.328+0000 7fe134b81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:40.341 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.328+0000 7fe134b81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:40.341 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.329+0000 7fe12ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:40.341 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.329+0000 7fe12ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:40.341 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.329+0000 7fe12ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:40.346 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.331+0000 7fe134b81640 10 monclient(hunting): handle_auth_done global_id 4149 payload 274 2023-11-02T08:35:40.346 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.331+0000 7fe134b81640 1 -- 172.21.15.151:0/3681626139 >> v2:172.21.15.178:3300/0 conn(0x7fe1300b41c0 msgr2=0x7fe1300a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:35:40.346 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.331+0000 7fe134b81640 1 --2- 172.21.15.151:0/3681626139 >> v2:172.21.15.178:3300/0 conn(0x7fe1300b41c0 0x7fe1300a8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:40.347 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.331+0000 7fe134b81640 1 -- 172.21.15.151:0/3681626139 >> v1:172.21.15.178:6789/0 conn(0x7fe13013e780 legacy=0x7fe1301416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:40.347 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.331+0000 7fe134b81640 10 monclient: _finish_hunting 0 2023-11-02T08:35:40.347 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.331+0000 7fe134b81640 1 monclient: found mon.noname-a 2023-11-02T08:35:40.347 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.331+0000 7fe134b81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:40.347 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.331+0000 7fe134b81640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:40.348 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.331+0000 7fe134b81640 1 -- 172.21.15.151:0/3681626139 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1300aa640 con 0x7fe130141db0 2023-11-02T08:35:40.348 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.333+0000 7fe134b81640 1 --2- 172.21.15.151:0/3681626139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe130141db0 0x7fe13013ebf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe1300669c0 tx=0x7fe12802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273ad3117103a319 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:40.348 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe12effd640 1 -- 172.21.15.151:0/3681626139 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12802c9f0 con 0x7fe130141db0 2023-11-02T08:35:40.348 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:40.348 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe12effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:40.349 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe12effd640 10 monclient: dump: 2023-11-02T08:35:40.349 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:40.349 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:40.349 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:40.349 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:40.349 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:40.350 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:40.350 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:40.350 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:40.350 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:40.350 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:40.351 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe12effd640 10 monclient: _finish_auth 0 2023-11-02T08:35:40.351 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe12effd640 10 monclient: _check_auth_tickets 2023-11-02T08:35:40.351 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe12effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:40.351 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe12effd640 1 -- 172.21.15.151:0/3681626139 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12802cb50 con 0x7fe130141db0 2023-11-02T08:35:40.351 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe1365e4640 5 monclient: authenticate success, global_id 4149 2023-11-02T08:35:40.352 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.339+0000 7fe1365e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:35:40.352 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.340+0000 7fe12effd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:40.352 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.340+0000 7fe1365e4640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:40.352 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.340+0000 7fe1365e4640 4 set_mon_vals no callback set 2023-11-02T08:35:40.352 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.340+0000 7fe12effd640 1 -- 172.21.15.151:0/3681626139 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1280343f0 con 0x7fe130141db0 2023-11-02T08:35:40.353 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.340+0000 7fe12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:40.353 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.340+0000 7fe12effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:40.353 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.340+0000 7fe12effd640 10 monclient: dump: 2023-11-02T08:35:40.353 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:40.353 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:40.353 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:40.354 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:40.354 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:40.354 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:40.354 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:40.354 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:40.355 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:40.355 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:40.355 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 10 monclient: shutdown 2023-11-02T08:35:40.355 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:40.355 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 1 -- 172.21.15.151:0/3681626139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe130141db0 msgr2=0x7fe13013ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:40.356 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 1 --2- 172.21.15.151:0/3681626139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe130141db0 0x7fe13013ebf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe1300669c0 tx=0x7fe12802be50 comp rx=0 tx=0).stop 2023-11-02T08:35:40.356 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 1 -- 172.21.15.151:0/3681626139 shutdown_connections 2023-11-02T08:35:40.356 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 1 --2- 172.21.15.151:0/3681626139 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe130141db0 0x7fe13013ebf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:40.356 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 1 --2- 172.21.15.151:0/3681626139 >> v2:172.21.15.178:3300/0 conn(0x7fe1300b41c0 0x7fe1300a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:40.356 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 1 -- 172.21.15.151:0/3681626139 >> 172.21.15.151:0/3681626139 conn(0x7fe130065e10 msgr2=0x7fe130066220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:40.357 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 1 -- 172.21.15.151:0/3681626139 shutdown_connections 2023-11-02T08:35:40.357 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.341+0000 7fe1365e4640 1 -- 172.21.15.151:0/3681626139 wait complete. 2023-11-02T08:35:40.357 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.342+0000 7fe1365e4640 10 monclient: build_initial_monmap 2023-11-02T08:35:40.357 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.342+0000 7fe1365e4640 10 monclient: monmap: 2023-11-02T08:35:40.357 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:35:40.357 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:40.358 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2023-11-02T08:35:40.358 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2023-11-02T08:35:40.358 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:40.358 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:40.358 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:40.359 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:40.359 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:40.359 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:40.359 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.342+0000 7fe1365e4640 1 librados: starting msgr at 2023-11-02T08:35:40.359 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.342+0000 7fe1365e4640 1 librados: starting objecter 2023-11-02T08:35:40.360 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.342+0000 7fe1365e4640 1 Processor -- start 2023-11-02T08:35:40.360 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.343+0000 7fe1365e4640 1 -- start start 2023-11-02T08:35:40.360 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.343+0000 7fe1365e4640 1 librados: setting wanted keys 2023-11-02T08:35:40.360 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.343+0000 7fe1365e4640 1 librados: calling monclient init 2023-11-02T08:35:40.360 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.343+0000 7fe1365e4640 10 monclient: init 2023-11-02T08:35:40.367 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.343+0000 7fe1365e4640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:40.367 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:35:40.367 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1300b41c0 0x7fe1300abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:40.368 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe1300b41c0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:40.368 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe13013e780 0x7fe1300ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:40.368 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe13013e780 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:40.368 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe130141db0 0x7fe1300b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:40.368 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe130141db0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:40.369 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:40.369 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1300b7570 con 0x7fe1300b41c0 2023-11-02T08:35:40.369 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:40.369 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1300ac3b0 con 0x7fe13013e780 2023-11-02T08:35:40.369 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:40.370 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1300ada60 con 0x7fe130141db0 2023-11-02T08:35:40.370 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:40.370 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe1365e4640 10 monclient(hunting): authenticate will time out at 1311.700197s 2023-11-02T08:35:40.370 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.344+0000 7fe12ffff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1300b41c0 0x7fe1300abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:40.370 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe134b81640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe130141db0 0x7fe1300b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:40.371 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12f7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe13013e780 0x7fe1300ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:40.371 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12ffff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1300b41c0 0x7fe1300abdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:36668/0 (socket says 172.21.15.151:36668) 2023-11-02T08:35:40.371 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12ffff640 1 -- 172.21.15.151:0/3637698195 learned_addr learned my addr 172.21.15.151:0/3637698195 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:35:40.371 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12ffff640 10 monclient(hunting): get_auth_request con 0x7fe1300b41c0 auth_method 0 2023-11-02T08:35:40.371 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:40.372 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12ffff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:40.372 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:40.372 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe134b81640 10 monclient(hunting): get_auth_request con 0x7fe130141db0 auth_method 0 2023-11-02T08:35:40.372 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe134b81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:40.372 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe134b81640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:40.373 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe134b81640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:40.373 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12f7fe640 10 monclient(hunting): get_auth_request con 0x7fe13013e780 auth_method 0 2023-11-02T08:35:40.373 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:40.373 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12f7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:40.373 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.345+0000 7fe12f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:40.373 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.346+0000 7fe12ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:40.374 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.346+0000 7fe12ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:40.374 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.346+0000 7fe12ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:40.374 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.346+0000 7fe134b81640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:40.374 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.346+0000 7fe134b81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:40.374 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.346+0000 7fe134b81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:40.375 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.346+0000 7fe12f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:40.375 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.346+0000 7fe12f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:40.375 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.346+0000 7fe12f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:40.375 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 10 monclient(hunting): handle_auth_done global_id 4152 payload 824 2023-11-02T08:35:40.375 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 1 -- 172.21.15.151:0/3637698195 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe130141db0 msgr2=0x7fe1300b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:40.376 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe130141db0 0x7fe1300b5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:40.376 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 1 -- 172.21.15.151:0/3637698195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe13013e780 msgr2=0x7fe1300ab400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:40.376 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe13013e780 0x7fe1300ab400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:40.376 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 10 monclient: _finish_hunting 0 2023-11-02T08:35:40.376 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 1 monclient: found mon.noname-a 2023-11-02T08:35:40.376 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:40.377 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:40.377 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.350+0000 7fe12ffff640 1 -- 172.21.15.151:0/3637698195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe128009b30 con 0x7fe1300b41c0 2023-11-02T08:35:40.377 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.354+0000 7fe12ffff640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1300b41c0 0x7fe1300abdc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe12000e9b0 tx=0x7fe12000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:40.377 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.354+0000 7fe12cff9640 10 client.?.objecter ms_handle_connect 0x7fe1300b41c0 2023-11-02T08:35:40.377 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.354+0000 7fe12cff9640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:40.378 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12cff9640 1 -- 172.21.15.151:0/3637698195 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120009670 con 0x7fe1300b41c0 2023-11-02T08:35:40.378 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:40.378 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:40.378 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12cff9640 10 monclient: dump: 2023-11-02T08:35:40.378 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:40.379 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:40.379 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:40.379 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:40.379 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:40.379 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:40.380 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:40.380 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:40.380 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:40.380 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:40.380 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12cff9640 10 monclient: _finish_auth 0 2023-11-02T08:35:40.380 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12cff9640 10 monclient: _check_auth_tickets 2023-11-02T08:35:40.381 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:40.381 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12cff9640 1 -- 172.21.15.151:0/3637698195 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12000c9a0 con 0x7fe1300b41c0 2023-11-02T08:35:40.381 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe1365e4640 5 monclient: authenticate success, global_id 4152 2023-11-02T08:35:40.381 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12cff9640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:40.382 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.360+0000 7fe12e7fc640 4 set_mon_vals no callback set 2023-11-02T08:35:40.382 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.361+0000 7fe1365e4640 10 monclient: _renew_subs 2023-11-02T08:35:40.382 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.361+0000 7fe12cff9640 1 -- 172.21.15.151:0/3637698195 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12000cd50 con 0x7fe1300b41c0 2023-11-02T08:35:40.382 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.361+0000 7fe1365e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:40.382 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.361+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1300b7c60 con 0x7fe1300b41c0 2023-11-02T08:35:40.383 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.362+0000 7fe12cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:40.383 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.362+0000 7fe1365e4640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:40.383 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.362+0000 7fe12cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:40.383 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.362+0000 7fe12cff9640 10 monclient: dump: 2023-11-02T08:35:40.383 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:40.384 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:40.384 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:40.384 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:40.384 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:40.384 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:40.385 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:40.385 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:40.385 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:40.385 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:40.385 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.362+0000 7fe1365e4640 10 monclient: _renew_subs 2023-11-02T08:35:40.385 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.362+0000 7fe1365e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:40.386 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.362+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1300aca20 con 0x7fe1300b41c0 2023-11-02T08:35:40.386 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.362+0000 7fe1365e4640 1 librados: init done 2023-11-02T08:35:40.386 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.363+0000 7fe1365e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:40.386 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.363+0000 7fe1365e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:40.386 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.363+0000 7fe1365e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:40.387 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.363+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe130067920 con 0x7fe1300b41c0 2023-11-02T08:35:40.399 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.396+0000 7fe12cff9640 1 -- 172.21.15.151:0/3637698195 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 88551+0+0 (secure 0 0 0) 0x7fe120013470 con 0x7fe1300b41c0 2023-11-02T08:35:40.400 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.397+0000 7fe12cff9640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe11c06bd60 0x7fe11c06e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:40.401 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.397+0000 7fe12f7fe640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe11c06bd60 0x7fe11c06e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:40.401 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.397+0000 7fe12f7fe640 10 monclient: get_auth_request con 0x7fe11c06bd60 auth_method 0 2023-11-02T08:35:40.401 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.398+0000 7fe12f7fe640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe11c06bd60 0x7fe11c06e220 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe130135810 tx=0x7fe124009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:40.402 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.398+0000 7fe12cff9640 10 client.4152.objecter ms_handle_connect 0x7fe11c06bd60 2023-11-02T08:35:40.844 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.840+0000 7fe12cff9640 1 -- 172.21.15.151:0/3637698195 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe120058170 con 0x7fe1300b41c0 2023-11-02T08:35:40.844 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.840+0000 7fe12cff9640 10 client.4152.objecter ms_dispatch 0x7fe130000b90 osd_map(10..10 src has 1..10) v4 2023-11-02T08:35:40.844 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.840+0000 7fe12cff9640 3 client.4152.objecter handle_osd_map got epochs [10,10] > 0 2023-11-02T08:35:40.845 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.840+0000 7fe12cff9640 3 client.4152.objecter handle_osd_map decoding full epoch 10 2023-11-02T08:35:40.845 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.840+0000 7fe12cff9640 20 client.4152.objecter dump_active .. 0 homeless 2023-11-02T08:35:40.845 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.840+0000 7fe12cff9640 1 -- 172.21.15.151:0/3637698195 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fe1200544b0 con 0x7fe1300b41c0 2023-11-02T08:35:40.845 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.840+0000 7fe12cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:40.845 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:40.840+0000 7fe12cff9640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:41.199 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.196+0000 7fe1365e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:41.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.196+0000 7fe1365e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:41.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.196+0000 7fe1365e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:35:41.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.196+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe13013dee0 con 0x7fe1300b41c0 2023-11-02T08:35:41.270 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.267+0000 7fe12cff9640 1 -- 172.21.15.151:0/3637698195 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe120017090 con 0x7fe1300b41c0 2023-11-02T08:35:41.271 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.267+0000 7fe12cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:41.271 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.267+0000 7fe12cff9640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:41.271 INFO:teuthology.misc.health.smithi151.stdout: 2023-11-02T08:35:41.272 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:35:37.853543+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:35:41.274 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.270+0000 7fe1365e4640 10 librados: watch_flush enter 2023-11-02T08:35:41.274 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.270+0000 7fe1365e4640 10 librados: watch_flush exit 2023-11-02T08:35:41.274 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.270+0000 7fe1365e4640 20 client.4152.objecter shutdown clearing up homeless session... 2023-11-02T08:35:41.275 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.270+0000 7fe1365e4640 10 client.4152.objecter successfully canceled tick 2023-11-02T08:35:41.275 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe11c06bd60 msgr2=0x7fe11c06e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:41.275 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe11c06bd60 0x7fe11c06e220 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe130135810 tx=0x7fe124009040 comp rx=0 tx=0).stop 2023-11-02T08:35:41.275 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 10 monclient: shutdown 2023-11-02T08:35:41.275 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:41.276 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1300b41c0 msgr2=0x7fe1300abdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:41.276 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1300b41c0 0x7fe1300abdc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe12000e9b0 tx=0x7fe12000ee70 comp rx=0 tx=0).stop 2023-11-02T08:35:41.276 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 shutdown_connections 2023-11-02T08:35:41.276 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe130141db0 0x7fe1300b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:41.277 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe11c06bd60 0x7fe11c06e220 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:41.277 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe13013e780 0x7fe1300ab400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:41.277 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 --2- 172.21.15.151:0/3637698195 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1300b41c0 0x7fe1300abdc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:41.277 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 >> 172.21.15.151:0/3637698195 conn(0x7fe130065e10 msgr2=0x7fe1301366d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:41.277 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 shutdown_connections 2023-11-02T08:35:41.277 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 -- 172.21.15.151:0/3637698195 wait complete. 2023-11-02T08:35:41.278 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:41.271+0000 7fe1365e4640 1 librados: shutdown 2023-11-02T08:35:41.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-02T08:35:46.121 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:07.828 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.192 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:07.900 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.203 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:07.936 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.266 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:07.999 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.350 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:08.060 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.391 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:08.110 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.410 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:08.120 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.450 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:08.176 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.456 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:08.176 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.460 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:08.199 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.508 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:08.234 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.525 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:08.265 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.533 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:08.254 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.593 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:08.313 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.715 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:08.448 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.719 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:35:46.710+0000 5dc2340 -1 Falling back to public interface 2023-11-02T08:35:46.783 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:08.516 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:46.811 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:35:46.802+0000 5dc2340 -1 Falling back to public interface 2023-11-02T08:35:46.992 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:35:46.982+0000 5dc2340 -1 Falling back to public interface 2023-11-02T08:35:47.002 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:35:46.994+0000 5dc2340 -1 Falling back to public interface 2023-11-02T08:35:47.091 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:35:47.083+0000 5dc2340 -1 Falling back to public interface 2023-11-02T08:35:47.097 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:35:47.086+0000 5dc2340 -1 Falling back to public interface 2023-11-02T08:35:47.153 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:08.860 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.171 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:35:47.163+0000 5dc2340 -1 Falling back to public interface 2023-11-02T08:35:47.340 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:09.072 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.370 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:35:47.361+0000 5dc2340 -1 Falling back to public interface 2023-11-02T08:35:47.485 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.195 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.489 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:09.208 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.506 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.213 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.557 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:09.296 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.573 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:09.300 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.653 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:09.374 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.687 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:09.419 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.812 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:09.531 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.831 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.542 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.865 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.572 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.872 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:09.604 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.909 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:09.648 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:47.918 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:09.645 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.003 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:09.724 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.018 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:09.757 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.040 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:09.773 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.094 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-02T08:35:48.160 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.868 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.185 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.183+0000 7fc51730a640 10 monclient: get_monmap_and_config 2023-11-02T08:35:48.185 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.183+0000 7fc51730a640 10 monclient: build_initial_monmap 2023-11-02T08:35:48.186 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.183+0000 7fc51730a640 10 monclient: monmap: 2023-11-02T08:35:48.186 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:35:48.186 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:48.187 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:35:48.183299+0000 2023-11-02T08:35:48.187 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:35:48.183299+0000 2023-11-02T08:35:48.187 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:48.187 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:48.187 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:48.188 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:48.188 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:48.188 INFO:teuthology.misc.health.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:48.188 INFO:teuthology.misc.health.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:48.188 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:48.191 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.183+0000 7fc51730a640 1 Processor -- start 2023-11-02T08:35:48.191 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 1 -- start start 2023-11-02T08:35:48.191 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 10 monclient: init 2023-11-02T08:35:48.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:48.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-11-02T08:35:48.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5100b41c0 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:48.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5100a8870 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:48.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc510141db0 0x7fc51014cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:48.193 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 10 monclient(hunting): picked mon.noname-c con 0x7fc510141db0 addr v2:172.21.15.178:3300/0 2023-11-02T08:35:48.193 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5100aa360 con 0x7fc5100b41c0 2023-11-02T08:35:48.193 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5100aa640 con 0x7fc5100a8870 2023-11-02T08:35:48.194 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:48.194 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5100a8d00 con 0x7fc510141db0 2023-11-02T08:35:48.194 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:48.194 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.184+0000 7fc51730a640 10 monclient(hunting): authenticate will time out at 1319.539557s 2023-11-02T08:35:48.194 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.185+0000 7fc5158a7640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc510141db0 0x7fc51014cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:48.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.185+0000 7fc5158a7640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc510141db0 0x7fc51014cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:41752/0 (socket says 172.21.15.151:41752) 2023-11-02T08:35:48.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.185+0000 7fc5158a7640 1 -- 172.21.15.151:0/99562765 learned_addr learned my addr 172.21.15.151:0/99562765 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:35:48.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.185+0000 7fc5158a7640 10 monclient(hunting): get_auth_request con 0x7fc510141db0 auth_method 0 2023-11-02T08:35:48.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.185+0000 7fc5158a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:48.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.185+0000 7fc5158a7640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:48.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.185+0000 7fc5158a7640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:48.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.186+0000 7fc5158a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:48.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.186+0000 7fc5158a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:48.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.186+0000 7fc5158a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:48.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc507fff640 1 -- 172.21.15.151:0/99562765 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365220066 0 0) 0x7fc5100aa360 con 0x7fc5100b41c0 2023-11-02T08:35:48.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc507fff640 10 monclient(hunting): _negotiate 2023-11-02T08:35:48.197 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc507fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:48.197 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc507fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:48.197 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc507fff640 10 monclient(hunting): my global_id is 4179 2023-11-02T08:35:48.197 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc507fff640 1 -- 172.21.15.151:0/99562765 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4fc003520 con 0x7fc5100b41c0 2023-11-02T08:35:48.197 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc5158a7640 10 monclient(hunting): handle_auth_done global_id 4219 payload 274 2023-11-02T08:35:48.198 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc5158a7640 1 -- 172.21.15.151:0/99562765 >> v1:172.21.15.178:6789/0 conn(0x7fc5100a8870 legacy=0x7fc5101416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:48.198 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc5158a7640 1 -- 172.21.15.151:0/99562765 >> v1:172.21.15.151:6789/0 conn(0x7fc5100b41c0 legacy=0x7fc51013e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:48.198 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc5158a7640 10 monclient: _finish_hunting 0 2023-11-02T08:35:48.198 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc5158a7640 1 monclient: found mon.noname-c 2023-11-02T08:35:48.198 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc5158a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:48.199 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc5158a7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:35:48.199 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.188+0000 7fc5158a7640 1 -- 172.21.15.151:0/99562765 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5100a8f00 con 0x7fc510141db0 2023-11-02T08:35:48.200 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.901 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.192+0000 7fc5158a7640 1 --2- 172.21.15.151:0/99562765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc510141db0 0x7fc51014cfc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc508009e80 tx=0x7fc50802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd2c72762d6f2d1d server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:48.201 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:09.916 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.205 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 1 -- 172.21.15.151:0/99562765 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50802ca70 con 0x7fc510141db0 2023-11-02T08:35:48.205 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:48.205 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:35:48.205 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 10 monclient: dump: 2023-11-02T08:35:48.206 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:48.206 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:48.206 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:48.206 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:48.206 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:48.207 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:48.207 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:48.207 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:48.207 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:48.207 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:48.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 10 monclient: _finish_auth 0 2023-11-02T08:35:48.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 10 monclient: _check_auth_tickets 2023-11-02T08:35:48.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:48.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 1 -- 172.21.15.151:0/99562765 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50802cbd0 con 0x7fc510141db0 2023-11-02T08:35:48.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:48.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 1 -- 172.21.15.151:0/99562765 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5080343f0 con 0x7fc510141db0 2023-11-02T08:35:48.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:48.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:48.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc507fff640 10 monclient: dump: 2023-11-02T08:35:48.210 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:48.210 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:48.210 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:48.210 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:48.211 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:48.211 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:48.211 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:48.211 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:48.211 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:48.212 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:48.212 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc51730a640 5 monclient: authenticate success, global_id 4219 2023-11-02T08:35:48.212 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc51730a640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:48.212 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.195+0000 7fc51730a640 4 set_mon_vals no callback set 2023-11-02T08:35:48.212 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.196+0000 7fc51730a640 10 monclient: shutdown 2023-11-02T08:35:48.213 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.196+0000 7fc51730a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:48.213 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.196+0000 7fc51730a640 1 -- 172.21.15.151:0/99562765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc510141db0 msgr2=0x7fc51014cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:48.213 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.196+0000 7fc51730a640 1 --2- 172.21.15.151:0/99562765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc510141db0 0x7fc51014cfc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc508009e80 tx=0x7fc50802bed0 comp rx=0 tx=0).stop 2023-11-02T08:35:48.213 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.196+0000 7fc51730a640 1 -- 172.21.15.151:0/99562765 shutdown_connections 2023-11-02T08:35:48.213 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.196+0000 7fc51730a640 1 --2- 172.21.15.151:0/99562765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc510141db0 0x7fc51014cfc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:48.214 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.196+0000 7fc51730a640 1 -- 172.21.15.151:0/99562765 >> 172.21.15.151:0/99562765 conn(0x7fc510065e10 msgr2=0x7fc510066220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:48.214 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.196+0000 7fc51730a640 1 -- 172.21.15.151:0/99562765 shutdown_connections 2023-11-02T08:35:48.214 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.196+0000 7fc51730a640 1 -- 172.21.15.151:0/99562765 wait complete. 2023-11-02T08:35:48.214 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.197+0000 7fc51730a640 10 monclient: build_initial_monmap 2023-11-02T08:35:48.214 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.197+0000 7fc51730a640 10 monclient: monmap: 2023-11-02T08:35:48.215 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:35:48.215 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:48.215 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2023-11-02T08:35:48.215 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2023-11-02T08:35:48.215 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:48.216 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:48.216 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:48.216 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:48.216 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:48.216 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:48.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 librados: starting msgr at 2023-11-02T08:35:48.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 librados: starting objecter 2023-11-02T08:35:48.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 Processor -- start 2023-11-02T08:35:48.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 -- start start 2023-11-02T08:35:48.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 librados: setting wanted keys 2023-11-02T08:35:48.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 librados: calling monclient init 2023-11-02T08:35:48.218 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient: init 2023-11-02T08:35:48.218 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:48.218 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-02T08:35:48.218 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5100a8870 0x7fc5100aad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:48.218 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient(hunting): picked mon.noname-c con 0x7fc5100a8870 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:48.219 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5100b41c0 0x7fc5101d9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:48.219 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient(hunting): picked mon.noname-b con 0x7fc5100b41c0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:48.219 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc510141db0 0x7fc510137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:48.219 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient(hunting): picked mon.noname-a con 0x7fc510141db0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:48.219 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:48.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100ac060 con 0x7fc510141db0 2023-11-02T08:35:48.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:48.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5101d97d0 con 0x7fc5100b41c0 2023-11-02T08:35:48.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:48.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100b71e0 con 0x7fc5100a8870 2023-11-02T08:35:48.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:48.221 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.198+0000 7fc51730a640 10 monclient(hunting): authenticate will time out at 1319.553635s 2023-11-02T08:35:48.221 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5158a7640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc510141db0 0x7fc510137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:48.221 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5148a5640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5100b41c0 0x7fc5101d9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:48.221 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5150a6640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5100a8870 0x7fc5100aad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:48.221 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5148a5640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5100b41c0 0x7fc5101d9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:41762/0 (socket says 172.21.15.151:41762) 2023-11-02T08:35:48.222 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5148a5640 1 -- 172.21.15.151:0/1176775656 learned_addr learned my addr 172.21.15.151:0/1176775656 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:35:48.222 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5148a5640 10 monclient(hunting): get_auth_request con 0x7fc5100b41c0 auth_method 0 2023-11-02T08:35:48.222 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5148a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:48.222 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5148a5640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:48.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5148a5640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:48.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5158a7640 10 monclient(hunting): get_auth_request con 0x7fc510141db0 auth_method 0 2023-11-02T08:35:48.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5158a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:48.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5158a7640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:48.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5158a7640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:48.224 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5150a6640 10 monclient(hunting): get_auth_request con 0x7fc5100a8870 auth_method 0 2023-11-02T08:35:48.224 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5150a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:48.224 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5150a6640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:48.224 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.199+0000 7fc5150a6640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:48.224 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.200+0000 7fc5148a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:48.224 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.200+0000 7fc5148a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:48.225 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.200+0000 7fc5148a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:48.225 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.200+0000 7fc5158a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:48.225 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.200+0000 7fc5158a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:48.225 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.200+0000 7fc5158a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:48.225 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.200+0000 7fc5150a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:48.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.200+0000 7fc5150a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:48.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.200+0000 7fc5150a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:48.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.202+0000 7fc5148a5640 10 monclient(hunting): handle_auth_done global_id 4222 payload 824 2023-11-02T08:35:48.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.202+0000 7fc5148a5640 1 -- 172.21.15.151:0/1176775656 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5100a8870 msgr2=0x7fc5100aad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:48.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.202+0000 7fc5148a5640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5100a8870 0x7fc5100aad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:48.227 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.203+0000 7fc5148a5640 1 -- 172.21.15.151:0/1176775656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc510141db0 msgr2=0x7fc510137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:48.227 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.203+0000 7fc5148a5640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc510141db0 0x7fc510137130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:48.227 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.203+0000 7fc5148a5640 10 monclient: _finish_hunting 0 2023-11-02T08:35:48.227 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.203+0000 7fc5148a5640 1 monclient: found mon.noname-b 2023-11-02T08:35:48.227 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.203+0000 7fc5148a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:48.228 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.203+0000 7fc5148a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:35:48.228 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.203+0000 7fc5148a5640 1 -- 172.21.15.151:0/1176775656 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc508009b30 con 0x7fc5100b41c0 2023-11-02T08:35:48.229 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.205+0000 7fc5148a5640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5100b41c0 0x7fc5101d9cf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc50000c450 tx=0x7fc50000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:48.229 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.205+0000 7fc505ffb640 10 client.?.objecter ms_handle_connect 0x7fc5100b41c0 2023-11-02T08:35:48.229 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.205+0000 7fc505ffb640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:48.230 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc505ffb640 1 -- 172.21.15.151:0/1176775656 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5000052d0 con 0x7fc5100b41c0 2023-11-02T08:35:48.230 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc505ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:48.230 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc505ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:35:48.230 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc505ffb640 10 monclient: dump: 2023-11-02T08:35:48.230 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:48.231 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:48.231 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:48.231 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:48.231 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:48.232 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:48.232 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:48.232 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:48.232 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:48.232 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:48.232 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc505ffb640 10 monclient: _finish_auth 0 2023-11-02T08:35:48.233 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc505ffb640 10 monclient: _check_auth_tickets 2023-11-02T08:35:48.233 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc505ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:48.233 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc51730a640 5 monclient: authenticate success, global_id 4222 2023-11-02T08:35:48.233 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc51730a640 10 monclient: _renew_subs 2023-11-02T08:35:48.233 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc51730a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:48.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5100b7410 con 0x7fc5100b41c0 2023-11-02T08:35:48.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc51730a640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:48.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc51730a640 10 monclient: _renew_subs 2023-11-02T08:35:48.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc51730a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:48.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5100b5000 con 0x7fc5100b41c0 2023-11-02T08:35:48.235 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.207+0000 7fc51730a640 1 librados: init done 2023-11-02T08:35:48.235 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.208+0000 7fc51730a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:48.235 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.208+0000 7fc51730a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:48.235 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.208+0000 7fc51730a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:48.235 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.208+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc510067920 con 0x7fc5100b41c0 2023-11-02T08:35:48.239 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:09.972 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.277 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.003 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.315 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:10.054 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.328 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.060 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.366 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:10.087 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.449 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.447+0000 7fc505ffb640 1 -- 172.21.15.151:0/1176775656 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5000011e0 con 0x7fc5100b41c0 2023-11-02T08:35:48.450 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.447+0000 7fc505ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:48.450 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.447+0000 7fc505ffb640 1 -- 172.21.15.151:0/1176775656 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc500007030 con 0x7fc5100b41c0 2023-11-02T08:35:48.450 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.447+0000 7fc505ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:48.450 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.447+0000 7fc505ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:35:48.451 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.447+0000 7fc505ffb640 10 monclient: dump: 2023-11-02T08:35:48.451 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:48.451 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:48.451 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:48.451 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:48.451 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:48.452 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:48.452 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:48.452 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:48.452 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:48.452 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:48.453 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.447+0000 7fc5077fe640 4 set_mon_vals no callback set 2023-11-02T08:35:48.453 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.157 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.454 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.448+0000 7fc505ffb640 1 -- 172.21.15.151:0/1176775656 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 4) v1 ==== 88551+0+0 (secure 0 0 0) 0x7fc500007210 con 0x7fc5100b41c0 2023-11-02T08:35:48.454 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.448+0000 7fc505ffb640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc4f406bd30 0x7fc4f406e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:48.454 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.448+0000 7fc5150a6640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc4f406bd30 0x7fc4f406e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:48.455 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.448+0000 7fc5150a6640 10 monclient: get_auth_request con 0x7fc4f406bd30 auth_method 0 2023-11-02T08:35:48.455 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.449+0000 7fc505ffb640 10 client.4222.objecter ms_handle_connect 0x7fc4f406bd30 2023-11-02T08:35:48.455 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.449+0000 7fc5150a6640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc4f406bd30 0x7fc4f406e1f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc50c000f60 tx=0x7fc50c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:48.462 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:10.181 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.464 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.462+0000 7fc505ffb640 1 -- 172.21.15.151:0/1176775656 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc5000580f0 con 0x7fc5100b41c0 2023-11-02T08:35:48.465 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.462+0000 7fc505ffb640 10 client.4222.objecter ms_dispatch 0x7fc510000b90 osd_map(10..10 src has 1..10) v4 2023-11-02T08:35:48.465 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.462+0000 7fc505ffb640 3 client.4222.objecter handle_osd_map got epochs [10,10] > 0 2023-11-02T08:35:48.465 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.462+0000 7fc505ffb640 3 client.4222.objecter handle_osd_map decoding full epoch 10 2023-11-02T08:35:48.465 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.462+0000 7fc505ffb640 20 client.4222.objecter dump_active .. 0 homeless 2023-11-02T08:35:48.466 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.462+0000 7fc505ffb640 1 -- 172.21.15.151:0/1176775656 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fc500054430 con 0x7fc5100b41c0 2023-11-02T08:35:48.466 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.462+0000 7fc505ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:48.466 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.462+0000 7fc505ffb640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:48.489 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.200 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.568 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.295 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.597 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:10.330 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.614 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:10.354 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.619 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.351 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.641 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.639+0000 7fc51730a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:48.641 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.639+0000 7fc51730a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:48.642 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.639+0000 7fc51730a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:35:48.642 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.639+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc510068b90 con 0x7fc5100b41c0 2023-11-02T08:35:48.655 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:10.376 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.657 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.654+0000 7fc505ffb640 1 -- 172.21.15.151:0/1176775656 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc500057a50 con 0x7fc5100b41c0 2023-11-02T08:35:48.657 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.654+0000 7fc505ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:48.658 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.654+0000 7fc505ffb640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:48.658 INFO:teuthology.misc.health.smithi151.stdout: 2023-11-02T08:35:48.659 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:35:37.853543+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:35:48.661 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 10 librados: watch_flush enter 2023-11-02T08:35:48.661 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 10 librados: watch_flush exit 2023-11-02T08:35:48.661 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 20 client.4222.objecter shutdown clearing up homeless session... 2023-11-02T08:35:48.661 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 10 client.4222.objecter successfully canceled tick 2023-11-02T08:35:48.662 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc4f406bd30 msgr2=0x7fc4f406e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:48.662 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc4f406bd30 0x7fc4f406e1f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc50c000f60 tx=0x7fc50c009040 comp rx=0 tx=0).stop 2023-11-02T08:35:48.662 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 10 monclient: shutdown 2023-11-02T08:35:48.663 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:48.663 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5100b41c0 msgr2=0x7fc5101d9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:48.663 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5100b41c0 0x7fc5101d9cf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc50000c450 tx=0x7fc50000c910 comp rx=0 tx=0).stop 2023-11-02T08:35:48.663 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 shutdown_connections 2023-11-02T08:35:48.663 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc510141db0 0x7fc510137130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:48.664 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc4f406bd30 0x7fc4f406e1f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:48.664 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5100b41c0 0x7fc5101d9cf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:48.664 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 --2- 172.21.15.151:0/1176775656 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5100a8870 0x7fc5100aad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:48.664 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 >> 172.21.15.151:0/1176775656 conn(0x7fc510065e10 msgr2=0x7fc51013a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:48.664 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 shutdown_connections 2023-11-02T08:35:48.665 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 -- 172.21.15.151:0/1176775656 wait complete. 2023-11-02T08:35:48.665 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:48.656+0000 7fc51730a640 1 librados: shutdown 2023-11-02T08:35:48.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-02T08:35:48.747 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.455 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.751 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:10.470 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.788 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.498 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.795 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:10.514 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.863 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.589 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.894 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:10.627 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.906 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.639 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.913 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:10.652 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:48.961 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:10.682 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.039 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.747 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.081 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.790 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.083 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:10.802 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.152 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.878 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.182 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:10.915 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.198 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.931 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.212 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:10.951 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.253 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:10.974 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.326 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:11.034 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.371 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:11.081 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.378 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:11.097 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.440 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:11.167 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.471 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:11.204 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.489 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:11.222 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.507 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:11.246 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.547 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:11.268 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.613 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:11.320 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.658 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:11.368 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.671 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:11.390 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.727 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:11.454 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.758 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:11.490 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.781 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:11.514 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.795 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:11.534 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.841 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:11.562 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:49.956 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:11.666 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.014 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:11.741 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.055 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:11.787 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.127 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:11.834 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.128 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:11.849 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.181 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:11.888 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.187 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:11.906 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.241 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:11.960 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.298 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:12.031 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.324 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:12.063 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.350 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:12.082 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.378 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:12.118 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.397 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:12.130 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.524 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:12.234 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.538 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:12.265 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.592 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:12.319 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.598 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:12.308 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.631 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:12.352 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.706 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:12.427 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.893 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:12.625 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:50.958 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:12.691 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:55.476 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-02T08:35:55.597 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.593+0000 7f78b385c640 10 monclient: get_monmap_and_config 2023-11-02T08:35:55.597 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.593+0000 7f78b385c640 10 monclient: build_initial_monmap 2023-11-02T08:35:55.598 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.593+0000 7f78b385c640 10 monclient: monmap: 2023-11-02T08:35:55.598 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:35:55.598 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:55.598 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:35:55.594970+0000 2023-11-02T08:35:55.599 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:35:55.594970+0000 2023-11-02T08:35:55.599 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:55.599 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:55.599 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:35:55.599 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:35:55.600 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:35:55.600 INFO:teuthology.misc.health.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:35:55.600 INFO:teuthology.misc.health.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:35:55.600 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:55.600 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.594+0000 7f78b385c640 1 Processor -- start 2023-11-02T08:35:55.602 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.594+0000 7f78b385c640 1 -- start start 2023-11-02T08:35:55.602 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.594+0000 7f78b385c640 10 monclient: init 2023-11-02T08:35:55.602 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:55.602 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-11-02T08:35:55.603 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78ac0b41c0 addr v1:172.21.15.178:6789/0 2023-11-02T08:35:55.603 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78ac0a8870 addr v1:172.21.15.151:6789/0 2023-11-02T08:35:55.603 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f78ac141db0 0x7f78ac14cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:55.603 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 10 monclient(hunting): picked mon.noname-a con 0x7f78ac141db0 addr v2:172.21.15.151:3300/0 2023-11-02T08:35:55.603 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac0aa360 con 0x7f78ac0a8870 2023-11-02T08:35:55.604 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac0aa640 con 0x7f78ac0b41c0 2023-11-02T08:35:55.604 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:55.604 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ac0a8d00 con 0x7f78ac141db0 2023-11-02T08:35:55.605 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:55.605 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.595+0000 7f78b385c640 10 monclient(hunting): authenticate will time out at 1326.951751s 2023-11-02T08:35:55.605 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.596+0000 7f78b15f8640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f78ac0b41c0 0x7f78ac13e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:60330/0 (socket says 172.21.15.151:60330) 2023-11-02T08:35:55.605 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.596+0000 7f78b15f8640 1 -- 172.21.15.151:0/632834137 learned_addr learned my addr 172.21.15.151:0/632834137 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:35:55.605 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.596+0000 7f78b1df9640 1 --2- 172.21.15.151:0/632834137 >> v2:172.21.15.151:3300/0 conn(0x7f78ac141db0 0x7f78ac14cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:55.607 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.597+0000 7f78b1df9640 10 monclient(hunting): get_auth_request con 0x7f78ac141db0 auth_method 0 2023-11-02T08:35:55.607 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.597+0000 7f78b1df9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:55.607 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.597+0000 7f78b1df9640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:55.607 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.597+0000 7f78b1df9640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:55.608 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.598+0000 7f78b1df9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:55.608 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.598+0000 7f78b1df9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:55.608 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.598+0000 7f78b1df9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:55.610 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.600+0000 7f78a3fff640 1 -- 172.21.15.151:0/632834137 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904566460 0 0) 0x7f78ac0aa640 con 0x7f78ac0b41c0 2023-11-02T08:35:55.610 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.600+0000 7f78a3fff640 10 monclient(hunting): _negotiate 2023-11-02T08:35:55.611 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.600+0000 7f78a3fff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:55.611 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.600+0000 7f78a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:55.611 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.600+0000 7f78a3fff640 10 monclient(hunting): my global_id is 4225 2023-11-02T08:35:55.611 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.600+0000 7f78a3fff640 1 -- 172.21.15.151:0/632834137 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7898003520 con 0x7f78ac0b41c0 2023-11-02T08:35:55.611 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.602+0000 7f78b1df9640 10 monclient(hunting): handle_auth_done global_id 4185 payload 274 2023-11-02T08:35:55.612 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.602+0000 7f78b1df9640 1 -- 172.21.15.151:0/632834137 >> v1:172.21.15.178:6789/0 conn(0x7f78ac0b41c0 legacy=0x7f78ac13e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:55.612 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.602+0000 7f78b1df9640 1 -- 172.21.15.151:0/632834137 >> v1:172.21.15.151:6789/0 conn(0x7f78ac0a8870 legacy=0x7f78ac1416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:55.612 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.602+0000 7f78b1df9640 10 monclient: _finish_hunting 0 2023-11-02T08:35:55.612 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.602+0000 7f78b1df9640 1 monclient: found mon.noname-a 2023-11-02T08:35:55.612 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.602+0000 7f78b1df9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:55.613 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.602+0000 7f78b1df9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:35:55.613 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.602+0000 7f78b1df9640 1 -- 172.21.15.151:0/632834137 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac0a8f00 con 0x7f78ac141db0 2023-11-02T08:35:55.613 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.606+0000 7f78b1df9640 1 --2- 172.21.15.151:0/632834137 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 0x7f78ac14cfc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009dc0 tx=0x7f78a402be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855ab52bf6f0370d server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:55.615 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 1 -- 172.21.15.151:0/632834137 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a402cbb0 con 0x7f78ac141db0 2023-11-02T08:35:55.615 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:55.616 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:35:55.616 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 10 monclient: dump: 2023-11-02T08:35:55.616 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:55.616 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:55.616 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:55.617 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:55.617 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:55.617 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:55.618 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:55.618 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:55.618 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:55.618 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:55.618 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 10 monclient: _finish_auth 0 2023-11-02T08:35:55.619 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:35:55.619 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:55.619 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 1 -- 172.21.15.151:0/632834137 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a402cd10 con 0x7f78ac141db0 2023-11-02T08:35:55.619 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78b385c640 5 monclient: authenticate success, global_id 4185 2023-11-02T08:35:55.619 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78b385c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:35:55.620 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:55.620 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78b385c640 10 monclient: get_monmap_and_config success 2023-11-02T08:35:55.620 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78b385c640 4 set_mon_vals no callback set 2023-11-02T08:35:55.620 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 1 -- 172.21.15.151:0/632834137 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4034660 con 0x7f78ac141db0 2023-11-02T08:35:55.620 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:55.620 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:35:55.621 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.611+0000 7f78a3fff640 10 monclient: dump: 2023-11-02T08:35:55.621 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:55.621 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:55.621 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:55.621 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:55.622 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:55.622 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:55.622 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:55.622 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:55.622 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:55.623 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:55.627 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.612+0000 7f78b385c640 10 monclient: shutdown 2023-11-02T08:35:55.627 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.612+0000 7f78b385c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:55.628 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.612+0000 7f78b385c640 1 -- 172.21.15.151:0/632834137 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 msgr2=0x7f78ac14cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:55.628 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.612+0000 7f78b385c640 1 --2- 172.21.15.151:0/632834137 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 0x7f78ac14cfc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009dc0 tx=0x7f78a402be00 comp rx=0 tx=0).stop 2023-11-02T08:35:55.628 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.612+0000 7f78b385c640 1 -- 172.21.15.151:0/632834137 shutdown_connections 2023-11-02T08:35:55.628 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.612+0000 7f78b385c640 1 --2- 172.21.15.151:0/632834137 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 0x7f78ac14cfc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:55.628 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.612+0000 7f78b385c640 1 -- 172.21.15.151:0/632834137 >> 172.21.15.151:0/632834137 conn(0x7f78ac065e10 msgr2=0x7f78ac066220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:55.629 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.612+0000 7f78b385c640 1 -- 172.21.15.151:0/632834137 shutdown_connections 2023-11-02T08:35:55.629 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.612+0000 7f78b385c640 1 -- 172.21.15.151:0/632834137 wait complete. 2023-11-02T08:35:55.629 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.613+0000 7f78b385c640 10 monclient: build_initial_monmap 2023-11-02T08:35:55.629 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.613+0000 7f78b385c640 10 monclient: monmap: 2023-11-02T08:35:55.629 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:35:55.630 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:35:55.630 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2023-11-02T08:35:55.630 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2023-11-02T08:35:55.630 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:35:55.630 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:55.630 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:35:55.631 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:35:55.631 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:35:55.631 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:55.631 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.614+0000 7f78b385c640 1 librados: starting msgr at 2023-11-02T08:35:55.631 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.614+0000 7f78b385c640 1 librados: starting objecter 2023-11-02T08:35:55.632 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.614+0000 7f78b385c640 1 Processor -- start 2023-11-02T08:35:55.632 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.614+0000 7f78b385c640 1 -- start start 2023-11-02T08:35:55.632 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.614+0000 7f78b385c640 1 librados: setting wanted keys 2023-11-02T08:35:55.632 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.614+0000 7f78b385c640 1 librados: calling monclient init 2023-11-02T08:35:55.632 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.614+0000 7f78b385c640 10 monclient: init 2023-11-02T08:35:55.633 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient: _reopen_session rank -1 2023-11-02T08:35:55.633 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-02T08:35:55.633 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78ac0a8870 0x7f78ac136650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:55.633 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient(hunting): picked mon.noname-c con 0x7f78ac0a8870 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:35:55.633 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78ac0b41c0 0x7f78ac0b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:55.634 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient(hunting): picked mon.noname-b con 0x7f78ac0b41c0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:35:55.634 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 0x7f78ac144b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:55.634 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient(hunting): picked mon.noname-a con 0x7f78ac141db0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:35:55.634 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:55.634 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac145750 con 0x7f78ac141db0 2023-11-02T08:35:55.634 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:55.635 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac144180 con 0x7f78ac0b41c0 2023-11-02T08:35:55.635 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:35:55.635 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac136b90 con 0x7f78ac0a8870 2023-11-02T08:35:55.635 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient(hunting): _renew_subs 2023-11-02T08:35:55.635 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b385c640 10 monclient(hunting): authenticate will time out at 1326.971288s 2023-11-02T08:35:55.636 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b15f8640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78ac0a8870 0x7f78ac136650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:55.636 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b0df7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78ac0b41c0 0x7f78ac0b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:55.636 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.615+0000 7f78b1df9640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 0x7f78ac144b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:55.636 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b1df9640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 0x7f78ac144b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:54072/0 (socket says 172.21.15.151:54072) 2023-11-02T08:35:55.636 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b1df9640 1 -- 172.21.15.151:0/2589595109 learned_addr learned my addr 172.21.15.151:0/2589595109 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:35:55.637 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b1df9640 10 monclient(hunting): get_auth_request con 0x7f78ac141db0 auth_method 0 2023-11-02T08:35:55.637 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b1df9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:55.637 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b1df9640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:55.637 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b1df9640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:55.637 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b15f8640 10 monclient(hunting): get_auth_request con 0x7f78ac0a8870 auth_method 0 2023-11-02T08:35:55.638 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b15f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:55.638 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b15f8640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:55.638 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b15f8640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:55.638 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b0df7640 10 monclient(hunting): get_auth_request con 0x7f78ac0b41c0 auth_method 0 2023-11-02T08:35:55.638 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b0df7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:35:55.638 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b0df7640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:35:55.639 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b0df7640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:35:55.639 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b1df9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:55.639 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b1df9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:55.639 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b1df9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:55.639 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b15f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:55.640 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b15f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:55.640 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.616+0000 7f78b15f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:55.640 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.617+0000 7f78b0df7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:35:55.640 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.617+0000 7f78b0df7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:35:55.640 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.617+0000 7f78b0df7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:35:55.641 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 10 monclient(hunting): handle_auth_done global_id 4166 payload 824 2023-11-02T08:35:55.641 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 1 -- 172.21.15.151:0/2589595109 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78ac0b41c0 msgr2=0x7f78ac0b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:55.641 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78ac0b41c0 0x7f78ac0b6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:55.641 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 1 -- 172.21.15.151:0/2589595109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 msgr2=0x7f78ac144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:55.641 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 0x7f78ac144b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:55.641 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 10 monclient: _finish_hunting 0 2023-11-02T08:35:55.642 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 1 monclient: found mon.noname-c 2023-11-02T08:35:55.642 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:35:55.642 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:35:55.642 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.620+0000 7f78b15f8640 1 -- 172.21.15.151:0/2589595109 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a4009a70 con 0x7f78ac0a8870 2023-11-02T08:35:55.642 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.623+0000 7f78b15f8640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78ac0a8870 0x7f78ac136650 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c430 tx=0x7f78a800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:55.643 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.623+0000 7f78a1ffb640 10 client.?.objecter ms_handle_connect 0x7f78ac0a8870 2023-11-02T08:35:55.643 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.623+0000 7f78a1ffb640 10 client.?.objecter resend_mon_ops 2023-11-02T08:35:55.644 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 1 -- 172.21.15.151:0/2589595109 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a8005ce0 con 0x7f78ac0a8870 2023-11-02T08:35:55.645 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:55.645 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:35:55.645 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 10 monclient: dump: 2023-11-02T08:35:55.645 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:55.645 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:55.646 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:55.646 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:55.646 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:55.646 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:55.646 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:55.647 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:55.647 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:55.647 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:55.647 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 10 monclient: _finish_auth 0 2023-11-02T08:35:55.647 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 10 monclient: _check_auth_tickets 2023-11-02T08:35:55.648 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:35:55.648 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 1 -- 172.21.15.151:0/2589595109 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a8005e40 con 0x7f78ac0a8870 2023-11-02T08:35:55.648 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78b385c640 5 monclient: authenticate success, global_id 4166 2023-11-02T08:35:55.648 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78b385c640 10 monclient: _renew_subs 2023-11-02T08:35:55.648 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78b385c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:35:55.648 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac132bd0 con 0x7f78ac0a8870 2023-11-02T08:35:55.649 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:35:55.649 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 1 -- 172.21.15.151:0/2589595109 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a80072d0 con 0x7f78ac0a8870 2023-11-02T08:35:55.649 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:35:55.649 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:35:55.649 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a1ffb640 10 monclient: dump: 2023-11-02T08:35:55.650 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:35:55.650 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:35:55.650 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:55.650 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:35:55.650 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:35:55.651 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:35:55.651 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:35:55.651 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:35:55.651 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:35:55.651 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:35:55.651 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.629+0000 7f78b385c640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:35:55.652 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.629+0000 7f78b385c640 10 monclient: _renew_subs 2023-11-02T08:35:55.652 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.628+0000 7f78a37fe640 4 set_mon_vals no callback set 2023-11-02T08:35:55.652 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.629+0000 7f78b385c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:35:55.652 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.629+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac0b71b0 con 0x7f78ac0a8870 2023-11-02T08:35:55.652 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.629+0000 7f78b385c640 1 librados: init done 2023-11-02T08:35:55.653 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.630+0000 7f78b385c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:55.653 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.630+0000 7f78b385c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:55.653 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.630+0000 7f78b385c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:35:55.653 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.630+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ac067920 con 0x7f78ac0a8870 2023-11-02T08:35:55.665 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.662+0000 7f78a1ffb640 1 -- 172.21.15.151:0/2589595109 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 4) v1 ==== 88551+0+0 (secure 0 0 0) 0x7f78a8007430 con 0x7f78ac0a8870 2023-11-02T08:35:55.666 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.663+0000 7f78a1ffb640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f789006bd30 0x7f789006e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:35:55.667 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.663+0000 7f78b0df7640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f789006bd30 0x7f789006e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:35:55.667 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.663+0000 7f78b0df7640 10 monclient: get_auth_request con 0x7f789006bd30 auth_method 0 2023-11-02T08:35:55.667 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.663+0000 7f78a1ffb640 10 client.4166.objecter ms_handle_connect 0x7f789006bd30 2023-11-02T08:35:55.668 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:55.663+0000 7f78b0df7640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f789006bd30 0x7f789006e1f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0aaf10 tx=0x7f789c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:35:56.026 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:17.734 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:56.102 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.098+0000 7f78a1ffb640 1 -- 172.21.15.151:0/2589595109 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f78a8057660 con 0x7f78ac0a8870 2023-11-02T08:35:56.102 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.099+0000 7f78a1ffb640 10 client.4166.objecter ms_dispatch 0x7f78ac000b90 osd_map(10..10 src has 1..10) v4 2023-11-02T08:35:56.103 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.099+0000 7f78a1ffb640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2023-11-02T08:35:56.103 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.099+0000 7f78a1ffb640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2023-11-02T08:35:56.103 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.099+0000 7f78a1ffb640 20 client.4166.objecter dump_active .. 0 homeless 2023-11-02T08:35:56.103 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.099+0000 7f78a1ffb640 1 -- 172.21.15.151:0/2589595109 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f78a80539a0 con 0x7f78ac0a8870 2023-11-02T08:35:56.103 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.099+0000 7f78a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:35:56.104 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.099+0000 7f78a1ffb640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:35:56.122 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:17.848 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:56.306 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:18.039 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:56.310 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:18.030 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:56.331 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:18.070 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:56.341 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:18.051 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:56.440 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:18.173 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:35:56.463 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.460+0000 7f78b385c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:56.464 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.460+0000 7f78b385c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:56.464 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.460+0000 7f78b385c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:35:56.464 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.460+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78ac0b2f30 con 0x7f78ac0a8870 2023-11-02T08:35:56.534 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.531+0000 7f78a1ffb640 1 -- 172.21.15.151:0/2589595109 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f78a8017020 con 0x7f78ac0a8870 2023-11-02T08:35:56.534 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.531+0000 7f78a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:35:56.535 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.531+0000 7f78a1ffb640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:35:56.535 INFO:teuthology.misc.health.smithi151.stdout: 2023-11-02T08:35:56.535 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:35:37.853543+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:35:56.537 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.533+0000 7f78b385c640 10 librados: watch_flush enter 2023-11-02T08:35:56.537 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.533+0000 7f78b385c640 10 librados: watch_flush exit 2023-11-02T08:35:56.537 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.533+0000 7f78b385c640 20 client.4166.objecter shutdown clearing up homeless session... 2023-11-02T08:35:56.538 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.533+0000 7f78b385c640 10 client.4166.objecter successfully canceled tick 2023-11-02T08:35:56.538 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.533+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f789006bd30 msgr2=0x7f789006e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:56.538 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.533+0000 7f78b385c640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f789006bd30 0x7f789006e1f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0aaf10 tx=0x7f789c009040 comp rx=0 tx=0).stop 2023-11-02T08:35:56.538 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 10 monclient: shutdown 2023-11-02T08:35:56.539 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:35:56.539 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78ac0a8870 msgr2=0x7f78ac136650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:35:56.539 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78ac0a8870 0x7f78ac136650 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c430 tx=0x7f78a800c8f0 comp rx=0 tx=0).stop 2023-11-02T08:35:56.539 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 shutdown_connections 2023-11-02T08:35:56.539 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f78ac141db0 0x7f78ac144b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:56.540 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f789006bd30 0x7f789006e1f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:56.540 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78ac0b41c0 0x7f78ac0b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:56.540 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 1 --2- 172.21.15.151:0/2589595109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f78ac0a8870 0x7f78ac136650 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:35:56.540 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 >> 172.21.15.151:0/2589595109 conn(0x7f78ac065e10 msgr2=0x7f78ac0b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:35:56.541 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.534+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 shutdown_connections 2023-11-02T08:35:56.541 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.535+0000 7f78b385c640 1 -- 172.21.15.151:0/2589595109 wait complete. 2023-11-02T08:35:56.541 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:35:56.535+0000 7f78b385c640 1 librados: shutdown 2023-11-02T08:35:56.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-02T08:35:56.554 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:18.275 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:00.744 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:22.452 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:00.749 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:22.456 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.015 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:22.742 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.019 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:22.746 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.216 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:22.949 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.220 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:22.953 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.255 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:22.965 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.258 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:22.969 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.302 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:23.042 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.307 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:23.047 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.582 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:23.301 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.585 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:23.305 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.586 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:23.319 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.590 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:23.323 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.749 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:23.470 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:01.753 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:23.474 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:03.355 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-02T08:36:03.468 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.465+0000 7f21e28bd640 10 monclient: get_monmap_and_config 2023-11-02T08:36:03.469 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.465+0000 7f21e28bd640 10 monclient: build_initial_monmap 2023-11-02T08:36:03.469 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.465+0000 7f21e28bd640 10 monclient: monmap: 2023-11-02T08:36:03.469 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:36:03.469 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:03.470 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:36:03.466512+0000 2023-11-02T08:36:03.470 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:36:03.466512+0000 2023-11-02T08:36:03.471 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:03.471 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:03.471 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:03.471 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:03.472 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:03.472 INFO:teuthology.misc.health.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:03.472 INFO:teuthology.misc.health.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:03.472 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:03.476 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 1 Processor -- start 2023-11-02T08:36:03.476 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 1 -- start start 2023-11-02T08:36:03.476 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 10 monclient: init 2023-11-02T08:36:03.477 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:03.477 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-11-02T08:36:03.477 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f21dc0b41c0 0x7f21dc0a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:03.477 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 10 monclient(hunting): picked mon.noname-c con 0x7f21dc0b41c0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:03.477 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 0x7f21dc13ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:03.478 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 10 monclient(hunting): picked mon.noname-b con 0x7f21dc13e780 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:03.478 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21dc1416a0 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:03.478 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21dc0aa360 con 0x7f21dc1416a0 2023-11-02T08:36:03.478 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.466+0000 7f21e28bd640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:03.478 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21e28bd640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc13f100 con 0x7f21dc0b41c0 2023-11-02T08:36:03.479 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21e28bd640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:03.479 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21e28bd640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0aa160 con 0x7f21dc13e780 2023-11-02T08:36:03.479 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21e28bd640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:03.480 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21e28bd640 10 monclient(hunting): authenticate will time out at 1334.822953s 2023-11-02T08:36:03.480 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21dbfff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f21dc0b41c0 0x7f21dc0a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:03.480 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21db7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 0x7f21dc13ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:03.480 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21e0e5a640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f21dc1416a0 0x7f21dc141b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:53338/0 (socket says 172.21.15.151:53338) 2023-11-02T08:36:03.481 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21e0e5a640 1 -- 172.21.15.151:0/1531158695 learned_addr learned my addr 172.21.15.151:0/1531158695 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:03.481 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.467+0000 7f21dbfff640 10 monclient(hunting): get_auth_request con 0x7f21dc0b41c0 auth_method 0 2023-11-02T08:36:03.481 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:03.481 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21dbfff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:03.482 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:03.482 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:03.482 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:03.483 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:03.483 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21db7fe640 10 monclient(hunting): get_auth_request con 0x7f21dc13e780 auth_method 0 2023-11-02T08:36:03.483 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:03.484 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21db7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:03.484 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.468+0000 7f21db7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:03.485 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.469+0000 7f21db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:03.485 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.469+0000 7f21db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:03.485 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.469+0000 7f21db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:03.485 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.470+0000 7f21dbfff640 10 monclient(hunting): handle_auth_done global_id 4231 payload 274 2023-11-02T08:36:03.486 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.470+0000 7f21dbfff640 1 -- 172.21.15.151:0/1531158695 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 msgr2=0x7f21dc13ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:03.486 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.470+0000 7f21dbfff640 1 --2- 172.21.15.151:0/1531158695 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 0x7f21dc13ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:03.487 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.470+0000 7f21dbfff640 1 -- 172.21.15.151:0/1531158695 >> v1:172.21.15.151:6789/0 conn(0x7f21dc1416a0 legacy=0x7f21dc141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:03.487 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.470+0000 7f21dbfff640 10 monclient: _finish_hunting 0 2023-11-02T08:36:03.487 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.470+0000 7f21dbfff640 1 monclient: found mon.noname-c 2023-11-02T08:36:03.488 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.470+0000 7f21dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:03.488 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.470+0000 7f21dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:03.488 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.470+0000 7f21dbfff640 1 -- 172.21.15.151:0/1531158695 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc0aa640 con 0x7f21dc0b41c0 2023-11-02T08:36:03.489 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.472+0000 7f21dbfff640 1 --2- 172.21.15.151:0/1531158695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc0b41c0 0x7f21dc0a8870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0097a0 tx=0x7f21cc02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc34bcdc63e63b7b server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:03.498 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 1 -- 172.21.15.151:0/1531158695 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc02f050 con 0x7f21dc0b41c0 2023-11-02T08:36:03.499 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:03.499 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:03.499 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 10 monclient: dump: 2023-11-02T08:36:03.499 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:03.500 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:03.500 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:03.500 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:03.500 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:03.500 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:03.501 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:03.501 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:03.501 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:03.502 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:03.502 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 10 monclient: _finish_auth 0 2023-11-02T08:36:03.502 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:03.502 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:03.502 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 1 -- 172.21.15.151:0/1531158695 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc031700 con 0x7f21dc0b41c0 2023-11-02T08:36:03.503 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:03.503 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 1 -- 172.21.15.151:0/1531158695 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc031a90 con 0x7f21dc0b41c0 2023-11-02T08:36:03.503 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:03.503 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:03.503 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21daffd640 10 monclient: dump: 2023-11-02T08:36:03.504 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:03.504 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:03.504 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:03.504 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:03.505 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:03.505 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:03.505 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:03.505 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:03.506 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:03.506 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:03.506 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 5 monclient: authenticate success, global_id 4231 2023-11-02T08:36:03.506 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:03.507 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 4 set_mon_vals no callback set 2023-11-02T08:36:03.507 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 10 monclient: shutdown 2023-11-02T08:36:03.507 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:03.507 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 1 -- 172.21.15.151:0/1531158695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc0b41c0 msgr2=0x7f21dc0a8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:03.507 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 1 --2- 172.21.15.151:0/1531158695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc0b41c0 0x7f21dc0a8870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0097a0 tx=0x7f21cc02b7a0 comp rx=0 tx=0).stop 2023-11-02T08:36:03.508 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 1 -- 172.21.15.151:0/1531158695 shutdown_connections 2023-11-02T08:36:03.508 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 1 --2- 172.21.15.151:0/1531158695 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 0x7f21dc13ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:03.508 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 1 --2- 172.21.15.151:0/1531158695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc0b41c0 0x7f21dc0a8870 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:03.508 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 1 -- 172.21.15.151:0/1531158695 >> 172.21.15.151:0/1531158695 conn(0x7f21dc065e10 msgr2=0x7f21dc066220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:03.508 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 1 -- 172.21.15.151:0/1531158695 shutdown_connections 2023-11-02T08:36:03.508 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.476+0000 7f21e28bd640 1 -- 172.21.15.151:0/1531158695 wait complete. 2023-11-02T08:36:03.509 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.477+0000 7f21e28bd640 10 monclient: build_initial_monmap 2023-11-02T08:36:03.509 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.477+0000 7f21e28bd640 10 monclient: monmap: 2023-11-02T08:36:03.509 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:36:03.509 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:03.509 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:03.510 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2023-11-02T08:36:03.510 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:03.511 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:03.511 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:03.511 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:03.511 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:03.511 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:03.512 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.477+0000 7f21e28bd640 1 librados: starting msgr at 2023-11-02T08:36:03.512 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.477+0000 7f21e28bd640 1 librados: starting objecter 2023-11-02T08:36:03.512 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.477+0000 7f21e28bd640 1 Processor -- start 2023-11-02T08:36:03.513 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.478+0000 7f21e28bd640 1 -- start start 2023-11-02T08:36:03.513 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.478+0000 7f21e28bd640 1 librados: setting wanted keys 2023-11-02T08:36:03.513 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.478+0000 7f21e28bd640 1 librados: calling monclient init 2023-11-02T08:36:03.513 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.478+0000 7f21e28bd640 10 monclient: init 2023-11-02T08:36:03.513 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.478+0000 7f21e28bd640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:03.514 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.478+0000 7f21e28bd640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:36:03.514 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.478+0000 7f21e28bd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21dc0b41c0 0x7f21dc13aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:03.514 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.478+0000 7f21e28bd640 10 monclient(hunting): picked mon.noname-a con 0x7f21dc0b41c0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:03.514 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.478+0000 7f21e28bd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 0x7f21dc13af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:03.515 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 10 monclient(hunting): picked mon.noname-c con 0x7f21dc13e780 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:03.516 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc1416a0 0x7f21dc1330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:03.516 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 10 monclient(hunting): picked mon.noname-b con 0x7f21dc1416a0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:03.516 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:03.516 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc13b490 con 0x7f21dc0b41c0 2023-11-02T08:36:03.516 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:03.517 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc13a4f0 con 0x7f21dc1416a0 2023-11-02T08:36:03.517 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:03.517 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1391c0 con 0x7f21dc13e780 2023-11-02T08:36:03.517 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:03.517 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21e28bd640 10 monclient(hunting): authenticate will time out at 1334.835660s 2023-11-02T08:36:03.518 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21dbfff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21dc0b41c0 0x7f21dc13aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:03.518 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.479+0000 7f21db7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 0x7f21dc13af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:03.518 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21e0e5a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc1416a0 0x7f21dc1330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:03.518 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21dbfff640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21dc0b41c0 0x7f21dc13aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:33398/0 (socket says 172.21.15.151:33398) 2023-11-02T08:36:03.518 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21dbfff640 1 -- 172.21.15.151:0/524699885 learned_addr learned my addr 172.21.15.151:0/524699885 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:03.519 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21dbfff640 10 monclient(hunting): get_auth_request con 0x7f21dc0b41c0 auth_method 0 2023-11-02T08:36:03.519 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:03.519 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21dbfff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:03.519 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:03.519 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21db7fe640 10 monclient(hunting): get_auth_request con 0x7f21dc13e780 auth_method 0 2023-11-02T08:36:03.520 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:03.520 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21db7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:03.520 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21db7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:03.520 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21e0e5a640 10 monclient(hunting): get_auth_request con 0x7f21dc1416a0 auth_method 0 2023-11-02T08:36:03.520 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21e0e5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:03.521 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21e0e5a640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:03.521 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21e0e5a640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:03.521 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:03.521 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.480+0000 7f21db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:03.521 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.481+0000 7f21db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:03.522 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.481+0000 7f21e0e5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:03.522 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.481+0000 7f21e0e5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:03.522 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.481+0000 7f21e0e5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:03.523 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.481+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:03.523 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.481+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:03.523 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.481+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:03.523 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 10 monclient(hunting): handle_auth_done global_id 4234 payload 824 2023-11-02T08:36:03.523 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 1 -- 172.21.15.151:0/524699885 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 msgr2=0x7f21dc13af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:03.524 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 0x7f21dc13af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:03.524 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 1 -- 172.21.15.151:0/524699885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21dc0b41c0 msgr2=0x7f21dc13aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:03.524 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21dc0b41c0 0x7f21dc13aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:03.525 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 10 monclient: _finish_hunting 0 2023-11-02T08:36:03.525 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 1 monclient: found mon.noname-b 2023-11-02T08:36:03.525 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:03.526 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:03.526 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.483+0000 7f21e0e5a640 1 -- 172.21.15.151:0/524699885 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc009400 con 0x7f21dc1416a0 2023-11-02T08:36:03.526 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.491+0000 7f21e0e5a640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc1416a0 0x7f21dc1330f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21d400caa0 tx=0x7f21d400cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:03.526 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.491+0000 7f21d8ff9640 10 client.?.objecter ms_handle_connect 0x7f21dc1416a0 2023-11-02T08:36:03.526 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.491+0000 7f21d8ff9640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:03.528 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 1 -- 172.21.15.151:0/524699885 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d400ab40 con 0x7f21dc1416a0 2023-11-02T08:36:03.528 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:03.529 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:03.529 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 10 monclient: dump: 2023-11-02T08:36:03.529 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:03.529 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:03.529 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:03.530 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:03.530 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:03.530 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:03.530 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:03.530 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:03.530 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:03.531 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:03.531 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 10 monclient: _finish_auth 0 2023-11-02T08:36:03.531 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 10 monclient: _check_auth_tickets 2023-11-02T08:36:03.531 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:03.531 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 1 -- 172.21.15.151:0/524699885 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d400aca0 con 0x7f21dc1416a0 2023-11-02T08:36:03.532 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:03.532 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 1 -- 172.21.15.151:0/524699885 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4012600 con 0x7f21dc1416a0 2023-11-02T08:36:03.532 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:03.532 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:03.532 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21d8ff9640 10 monclient: dump: 2023-11-02T08:36:03.533 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:03.533 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:03.533 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:03.533 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:03.533 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:03.533 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:03.534 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:03.534 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:03.534 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:03.534 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:03.534 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21e28bd640 5 monclient: authenticate success, global_id 4234 2023-11-02T08:36:03.535 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21e28bd640 10 monclient: _renew_subs 2023-11-02T08:36:03.535 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21da7fc640 4 set_mon_vals no callback set 2023-11-02T08:36:03.535 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21e28bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:03.535 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc137440 con 0x7f21dc1416a0 2023-11-02T08:36:03.535 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21e28bd640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:03.536 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21e28bd640 10 monclient: _renew_subs 2023-11-02T08:36:03.536 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21e28bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:03.536 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc133810 con 0x7f21dc1416a0 2023-11-02T08:36:03.536 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.497+0000 7f21e28bd640 1 librados: init done 2023-11-02T08:36:03.536 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.498+0000 7f21e28bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:03.537 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.498+0000 7f21e28bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:03.537 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.498+0000 7f21e28bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:03.537 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.498+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21dc137630 con 0x7f21dc1416a0 2023-11-02T08:36:03.537 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.522+0000 7f21d8ff9640 1 -- 172.21.15.151:0/524699885 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 4) v1 ==== 88551+0+0 (secure 0 0 0) 0x7f21d4007070 con 0x7f21dc1416a0 2023-11-02T08:36:03.537 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.523+0000 7f21d8ff9640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f21c806bd80 0x7f21c806e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:03.538 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.523+0000 7f21dbfff640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f21c806bd80 0x7f21c806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:03.538 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.523+0000 7f21dbfff640 10 monclient: get_auth_request con 0x7f21c806bd80 auth_method 0 2023-11-02T08:36:03.538 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.524+0000 7f21dbfff640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f21c806bd80 0x7f21c806e240 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f21cc02b7e0 tx=0x7f21cc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:03.538 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.524+0000 7f21d8ff9640 10 client.4234.objecter ms_handle_connect 0x7f21c806bd80 2023-11-02T08:36:03.538 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.524+0000 7f21d8ff9640 1 -- 172.21.15.151:0/524699885 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f21d4057f90 con 0x7f21dc1416a0 2023-11-02T08:36:03.539 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.524+0000 7f21d8ff9640 10 client.4234.objecter ms_dispatch 0x7f21dc000b90 osd_map(10..10 src has 1..10) v4 2023-11-02T08:36:03.539 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.524+0000 7f21d8ff9640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2023-11-02T08:36:03.539 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.524+0000 7f21d8ff9640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2023-11-02T08:36:03.539 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.524+0000 7f21d8ff9640 20 client.4234.objecter dump_active .. 0 homeless 2023-11-02T08:36:03.896 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.893+0000 7f21d8ff9640 1 -- 172.21.15.151:0/524699885 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f21d40542d0 con 0x7f21dc1416a0 2023-11-02T08:36:03.896 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.893+0000 7f21d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:03.896 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:03.893+0000 7f21d8ff9640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:04.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.213+0000 7f21e28bd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:04.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.213+0000 7f21e28bd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:04.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.213+0000 7f21e28bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:04.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.213+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21dc13d850 con 0x7f21dc1416a0 2023-11-02T08:36:04.240 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.237+0000 7f21d8ff9640 1 -- 172.21.15.151:0/524699885 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f21d40578f0 con 0x7f21dc1416a0 2023-11-02T08:36:04.241 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.237+0000 7f21d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:04.241 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.237+0000 7f21d8ff9640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:04.241 INFO:teuthology.misc.health.smithi151.stdout: 2023-11-02T08:36:04.242 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:35:37.853543+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:36:04.243 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 10 librados: watch_flush enter 2023-11-02T08:36:04.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 10 librados: watch_flush exit 2023-11-02T08:36:04.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 20 client.4234.objecter shutdown clearing up homeless session... 2023-11-02T08:36:04.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 10 client.4234.objecter successfully canceled tick 2023-11-02T08:36:04.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f21c806bd80 msgr2=0x7f21c806e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:04.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f21c806bd80 0x7f21c806e240 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f21cc02b7e0 tx=0x7f21cc008030 comp rx=0 tx=0).stop 2023-11-02T08:36:04.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 10 monclient: shutdown 2023-11-02T08:36:04.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:04.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc1416a0 msgr2=0x7f21dc1330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:04.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc1416a0 0x7f21dc1330f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21d400caa0 tx=0x7f21d400cf60 comp rx=0 tx=0).stop 2023-11-02T08:36:04.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 shutdown_connections 2023-11-02T08:36:04.246 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f21c806bd80 0x7f21c806e240 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:04.246 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21dc1416a0 0x7f21dc1330f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:04.246 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f21dc13e780 0x7f21dc13af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:04.247 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 --2- 172.21.15.151:0/524699885 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f21dc0b41c0 0x7f21dc13aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:04.247 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 >> 172.21.15.151:0/524699885 conn(0x7f21dc065e10 msgr2=0x7f21dc136990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:04.247 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 shutdown_connections 2023-11-02T08:36:04.247 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 -- 172.21.15.151:0/524699885 wait complete. 2023-11-02T08:36:04.247 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:04.240+0000 7f21e28bd640 1 librados: shutdown 2023-11-02T08:36:04.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-02T08:36:05.462 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:27.170 86123== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:05.483 INFO:tasks.ceph.osd.0.smithi151.stderr:==00:00:00:27.222 86098== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:05.505 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:27.231 86121== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:05.810 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:27.520 86122== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:05.990 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:27.722 86120== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:06.240 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:00:27.960 86100== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:06.259 INFO:tasks.ceph.osd.3.smithi151.stderr:==00:00:00:27.992 86099== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:06.698 INFO:tasks.ceph.osd.1.smithi151.stderr:==00:00:00:28.418 86101== Warning: unimplemented fcntl command: 1038 2023-11-02T08:36:11.061 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-02T08:36:11.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.197+0000 7fa1f5c5f640 10 monclient: get_monmap_and_config 2023-11-02T08:36:11.201 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.197+0000 7fa1f5c5f640 10 monclient: build_initial_monmap 2023-11-02T08:36:11.201 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.197+0000 7fa1f5c5f640 10 monclient: monmap: 2023-11-02T08:36:11.201 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:36:11.201 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:11.202 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:36:11.198326+0000 2023-11-02T08:36:11.202 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:36:11.198326+0000 2023-11-02T08:36:11.202 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:11.202 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:11.202 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:11.203 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:11.203 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:11.203 INFO:teuthology.misc.health.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:11.203 INFO:teuthology.misc.health.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:11.203 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:11.206 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.198+0000 7fa1f5c5f640 1 Processor -- start 2023-11-02T08:36:11.206 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.198+0000 7fa1f5c5f640 1 -- start start 2023-11-02T08:36:11.207 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.198+0000 7fa1f5c5f640 10 monclient: init 2023-11-02T08:36:11.207 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.199+0000 7fa1f5c5f640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:11.207 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.199+0000 7fa1f5c5f640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-11-02T08:36:11.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.199+0000 7fa1f5c5f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1f00b41c0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:11.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.199+0000 7fa1f5c5f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1f00a8870 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:11.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.199+0000 7fa1f5c5f640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f0141db0 0x7fa1f014cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:11.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.199+0000 7fa1f5c5f640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f0141db0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:11.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.199+0000 7fa1f5c5f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f00aa360 con 0x7fa1f00a8870 2023-11-02T08:36:11.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1effff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f0141db0 0x7fa1f014cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:11.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1ef7fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fa1f00b41c0 0x7fa1f013e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:33932/0 (socket says 172.21.15.151:33932) 2023-11-02T08:36:11.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1ef7fe640 1 -- 172.21.15.151:0/3003250946 learned_addr learned my addr 172.21.15.151:0/3003250946 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:11.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/3003250946 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f00aa640 con 0x7fa1f00b41c0 2023-11-02T08:36:11.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1f5c5f640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:11.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/3003250946 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f00a8d00 con 0x7fa1f0141db0 2023-11-02T08:36:11.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1f5c5f640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:11.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1f5c5f640 10 monclient(hunting): authenticate will time out at 1342.556566s 2023-11-02T08:36:11.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1effff640 10 monclient(hunting): get_auth_request con 0x7fa1f0141db0 auth_method 0 2023-11-02T08:36:11.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:11.211 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1effff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:11.211 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.200+0000 7fa1effff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:11.211 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.201+0000 7fa1effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:11.211 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.201+0000 7fa1effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:11.211 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.201+0000 7fa1effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:11.213 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.203+0000 7fa1ee7fc640 1 -- 172.21.15.151:0/3003250946 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965545254 0 0) 0x7fa1f00aa640 con 0x7fa1f00b41c0 2023-11-02T08:36:11.213 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.203+0000 7fa1ee7fc640 10 monclient(hunting): _negotiate 2023-11-02T08:36:11.213 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.203+0000 7fa1ee7fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:11.213 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.203+0000 7fa1ee7fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:11.214 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.203+0000 7fa1ee7fc640 10 monclient(hunting): my global_id is 4237 2023-11-02T08:36:11.214 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.203+0000 7fa1ee7fc640 1 -- 172.21.15.151:0/3003250946 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8003520 con 0x7fa1f00b41c0 2023-11-02T08:36:11.214 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.204+0000 7fa1effff640 10 monclient(hunting): handle_auth_done global_id 4175 payload 274 2023-11-02T08:36:11.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.204+0000 7fa1effff640 1 -- 172.21.15.151:0/3003250946 >> v1:172.21.15.178:6789/0 conn(0x7fa1f00b41c0 legacy=0x7fa1f013e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:11.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.204+0000 7fa1effff640 1 -- 172.21.15.151:0/3003250946 >> v1:172.21.15.151:6789/0 conn(0x7fa1f00a8870 legacy=0x7fa1f01416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:11.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.204+0000 7fa1effff640 10 monclient: _finish_hunting 0 2023-11-02T08:36:11.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.204+0000 7fa1effff640 1 monclient: found mon.noname-b 2023-11-02T08:36:11.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.204+0000 7fa1effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:11.216 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.204+0000 7fa1effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:11.216 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.204+0000 7fa1effff640 1 -- 172.21.15.151:0/3003250946 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f00a8f00 con 0x7fa1f0141db0 2023-11-02T08:36:11.216 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.209+0000 7fa1effff640 1 --2- 172.21.15.151:0/3003250946 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f0141db0 0x7fa1f014cfc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8009df0 tx=0x7fa1e802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e87019008d3d6a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:11.222 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 1 -- 172.21.15.151:0/3003250946 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e802cbb0 con 0x7fa1f0141db0 2023-11-02T08:36:11.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:11.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:11.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 10 monclient: dump: 2023-11-02T08:36:11.223 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:11.224 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:11.224 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:11.224 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:11.224 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:11.224 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:11.224 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:11.225 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:11.225 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:11.225 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:11.225 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:11.225 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:11.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:11.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 1 -- 172.21.15.151:0/3003250946 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e802cd10 con 0x7fa1f0141db0 2023-11-02T08:36:11.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1f5c5f640 5 monclient: authenticate success, global_id 4175 2023-11-02T08:36:11.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1f5c5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:11.226 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:11.227 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1f5c5f640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:11.227 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1f5c5f640 4 set_mon_vals no callback set 2023-11-02T08:36:11.227 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 1 -- 172.21.15.151:0/3003250946 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8034660 con 0x7fa1f0141db0 2023-11-02T08:36:11.227 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:11.228 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:11.228 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.219+0000 7fa1ee7fc640 10 monclient: dump: 2023-11-02T08:36:11.228 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:11.228 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:11.228 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:11.228 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:11.229 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:11.229 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:11.229 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:11.229 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:11.229 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:11.230 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:11.233 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.220+0000 7fa1f5c5f640 10 monclient: shutdown 2023-11-02T08:36:11.233 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.220+0000 7fa1f5c5f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:11.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.220+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/3003250946 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f0141db0 msgr2=0x7fa1f014cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:11.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.220+0000 7fa1f5c5f640 1 --2- 172.21.15.151:0/3003250946 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f0141db0 0x7fa1f014cfc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8009df0 tx=0x7fa1e802be00 comp rx=0 tx=0).stop 2023-11-02T08:36:11.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.220+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/3003250946 shutdown_connections 2023-11-02T08:36:11.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.220+0000 7fa1f5c5f640 1 --2- 172.21.15.151:0/3003250946 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f0141db0 0x7fa1f014cfc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:11.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.220+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/3003250946 >> 172.21.15.151:0/3003250946 conn(0x7fa1f0065e10 msgr2=0x7fa1f0066220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:11.235 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.221+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/3003250946 shutdown_connections 2023-11-02T08:36:11.235 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.221+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/3003250946 wait complete. 2023-11-02T08:36:11.235 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.222+0000 7fa1f5c5f640 10 monclient: build_initial_monmap 2023-11-02T08:36:11.235 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.222+0000 7fa1f5c5f640 10 monclient: monmap: 2023-11-02T08:36:11.235 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:36:11.236 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:11.236 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:11.236 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2023-11-02T08:36:11.236 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:11.236 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:11.237 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:11.237 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:11.237 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:11.237 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:11.237 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.222+0000 7fa1f5c5f640 1 librados: starting msgr at 2023-11-02T08:36:11.238 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.222+0000 7fa1f5c5f640 1 librados: starting objecter 2023-11-02T08:36:11.238 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.223+0000 7fa1f5c5f640 1 Processor -- start 2023-11-02T08:36:11.238 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.223+0000 7fa1f5c5f640 1 -- start start 2023-11-02T08:36:11.238 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.223+0000 7fa1f5c5f640 1 librados: setting wanted keys 2023-11-02T08:36:11.238 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.223+0000 7fa1f5c5f640 1 librados: calling monclient init 2023-11-02T08:36:11.239 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.223+0000 7fa1f5c5f640 10 monclient: init 2023-11-02T08:36:11.239 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.224+0000 7fa1f5c5f640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:11.239 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.224+0000 7fa1f5c5f640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-02T08:36:11.239 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.224+0000 7fa1f5c5f640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f00a8870 0x7fa1f00b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:11.239 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f00a8870 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:11.239 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1f00b41c0 0x7fa1f01d9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:11.240 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f00b41c0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:11.240 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1f0141db0 0x7fa1f01454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:11.240 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f0141db0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:11.240 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:11.240 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f0144f30 con 0x7fa1f00b41c0 2023-11-02T08:36:11.241 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:11.241 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f01da520 con 0x7fa1f0141db0 2023-11-02T08:36:11.241 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1ef7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f00a8870 0x7fa1f00b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:11.241 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:11.241 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f01459e0 con 0x7fa1f00a8870 2023-11-02T08:36:11.241 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:11.242 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.225+0000 7fa1f5c5f640 10 monclient(hunting): authenticate will time out at 1342.581785s 2023-11-02T08:36:11.242 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1effff640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1f0141db0 0x7fa1f01454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:11.242 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1eeffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1f00b41c0 0x7fa1f01d9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:11.242 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1ef7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f00a8870 0x7fa1f00b75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:39742/0 (socket says 172.21.15.151:39742) 2023-11-02T08:36:11.242 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1ef7fe640 1 -- 172.21.15.151:0/2375962654 learned_addr learned my addr 172.21.15.151:0/2375962654 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:11.243 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa1f00a8870 auth_method 0 2023-11-02T08:36:11.243 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:11.243 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1ef7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:11.243 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1ef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:11.243 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1eeffd640 10 monclient(hunting): get_auth_request con 0x7fa1f00b41c0 auth_method 0 2023-11-02T08:36:11.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:11.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1eeffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:11.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1eeffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:11.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:11.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:11.244 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.226+0000 7fa1ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:11.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.227+0000 7fa1eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:11.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.227+0000 7fa1eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:11.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.227+0000 7fa1eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:11.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.228+0000 7fa1effff640 10 monclient(hunting): get_auth_request con 0x7fa1f0141db0 auth_method 0 2023-11-02T08:36:11.245 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.228+0000 7fa1effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:11.246 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.228+0000 7fa1effff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:11.246 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.228+0000 7fa1effff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:11.249 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 10 monclient(hunting): handle_auth_done global_id 4178 payload 824 2023-11-02T08:36:11.249 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 1 -- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1f0141db0 msgr2=0x7fa1f01454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:11.249 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1f0141db0 0x7fa1f01454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:11.249 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 1 -- 172.21.15.151:0/2375962654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1f00b41c0 msgr2=0x7fa1f01d9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:11.250 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1f00b41c0 0x7fa1f01d9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:11.250 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 10 monclient: _finish_hunting 0 2023-11-02T08:36:11.250 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 1 monclient: found mon.noname-c 2023-11-02T08:36:11.250 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:11.251 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:11.251 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1ef7fe640 1 -- 172.21.15.151:0/2375962654 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e8009a70 con 0x7fa1f00a8870 2023-11-02T08:36:11.251 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.231+0000 7fa1effff640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1f0141db0 0x7fa1f01454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-02T08:36:11.251 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.235+0000 7fa1ef7fe640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f00a8870 0x7fa1f00b75c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000c530 tx=0x7fa1e000c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:11.251 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.235+0000 7fa1d7fff640 10 client.?.objecter ms_handle_connect 0x7fa1f00a8870 2023-11-02T08:36:11.252 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.235+0000 7fa1d7fff640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:11.252 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.240+0000 7fa1d7fff640 1 -- 172.21.15.151:0/2375962654 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0005ce0 con 0x7fa1f00a8870 2023-11-02T08:36:11.252 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.240+0000 7fa1d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:11.252 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:11.252 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 10 monclient: dump: 2023-11-02T08:36:11.252 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:11.253 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:11.253 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:11.253 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:11.253 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:11.253 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:11.254 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:11.254 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:11.254 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:11.254 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:11.254 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 10 monclient: _finish_auth 0 2023-11-02T08:36:11.255 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:11.255 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:11.255 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 1 -- 172.21.15.151:0/2375962654 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e0005e40 con 0x7fa1f00a8870 2023-11-02T08:36:11.255 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1f5c5f640 5 monclient: authenticate success, global_id 4178 2023-11-02T08:36:11.255 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1f5c5f640 10 monclient: _renew_subs 2023-11-02T08:36:11.256 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1f5c5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:11.256 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f01439e0 con 0x7fa1f00a8870 2023-11-02T08:36:11.256 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:11.256 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 1 -- 172.21.15.151:0/2375962654 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00072b0 con 0x7fa1f00a8870 2023-11-02T08:36:11.256 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:11.256 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:11.257 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1d7fff640 10 monclient: dump: 2023-11-02T08:36:11.257 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:11.257 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:11.257 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:11.257 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:11.258 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:11.258 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:11.258 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:11.258 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:11.258 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:11.259 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:11.259 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1f5c5f640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:11.259 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1f5c5f640 10 monclient: _renew_subs 2023-11-02T08:36:11.259 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1f5c5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:11.259 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f0144800 con 0x7fa1f00a8870 2023-11-02T08:36:11.260 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1f5c5f640 1 librados: init done 2023-11-02T08:36:11.260 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.241+0000 7fa1edffb640 4 set_mon_vals no callback set 2023-11-02T08:36:11.260 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.242+0000 7fa1f5c5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:11.260 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.242+0000 7fa1f5c5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:11.260 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.242+0000 7fa1f5c5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:11.260 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.242+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f00abcf0 con 0x7fa1f00a8870 2023-11-02T08:36:11.269 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.265+0000 7fa1d7fff640 1 -- 172.21.15.151:0/2375962654 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 4) v1 ==== 88551+0+0 (secure 0 0 0) 0x7fa1e0007410 con 0x7fa1f00a8870 2023-11-02T08:36:11.269 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.265+0000 7fa1d7fff640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fa1dc06bdb0 0x7fa1dc06e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:11.269 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.265+0000 7fa1eeffd640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fa1dc06bdb0 0x7fa1dc06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:11.269 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.265+0000 7fa1eeffd640 10 monclient: get_auth_request con 0x7fa1dc06bdb0 auth_method 0 2023-11-02T08:36:11.270 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.266+0000 7fa1d7fff640 10 client.4178.objecter ms_handle_connect 0x7fa1dc06bdb0 2023-11-02T08:36:11.270 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.266+0000 7fa1eeffd640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fa1dc06bdb0 0x7fa1dc06e270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00b7cf0 tx=0x7fa1e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:11.520 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.515+0000 7fa1d7fff640 1 -- 172.21.15.151:0/2375962654 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa1e0057740 con 0x7fa1f00a8870 2023-11-02T08:36:11.520 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.515+0000 7fa1d7fff640 10 client.4178.objecter ms_dispatch 0x7fa1f0000b90 osd_map(10..10 src has 1..10) v4 2023-11-02T08:36:11.521 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.515+0000 7fa1d7fff640 3 client.4178.objecter handle_osd_map got epochs [10,10] > 0 2023-11-02T08:36:11.521 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.515+0000 7fa1d7fff640 3 client.4178.objecter handle_osd_map decoding full epoch 10 2023-11-02T08:36:11.521 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.515+0000 7fa1d7fff640 20 client.4178.objecter dump_active .. 0 homeless 2023-11-02T08:36:11.521 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.515+0000 7fa1d7fff640 1 -- 172.21.15.151:0/2375962654 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fa1e0053a80 con 0x7fa1f00a8870 2023-11-02T08:36:11.522 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.515+0000 7fa1d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:11.522 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.515+0000 7fa1d7fff640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:11.716 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.712+0000 7fa1f5c5f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:11.716 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.712+0000 7fa1f5c5f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:11.716 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.712+0000 7fa1f5c5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:11.716 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.712+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1f01451c0 con 0x7fa1f00a8870 2023-11-02T08:36:11.744 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.741+0000 7fa1d7fff640 1 -- 172.21.15.151:0/2375962654 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa1f01451c0 con 0x7fa1f00a8870 2023-11-02T08:36:11.745 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.741+0000 7fa1d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:11.745 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.741+0000 7fa1d7fff640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:11.745 INFO:teuthology.misc.health.smithi151.stdout: 2023-11-02T08:36:11.746 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:35:37.853543+0000","last_up_change":"0.000000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:36:11.747 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 10 librados: watch_flush enter 2023-11-02T08:36:11.747 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 10 librados: watch_flush exit 2023-11-02T08:36:11.747 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 20 client.4178.objecter shutdown clearing up homeless session... 2023-11-02T08:36:11.747 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 10 client.4178.objecter successfully canceled tick 2023-11-02T08:36:11.748 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fa1dc06bdb0 msgr2=0x7fa1dc06e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:11.749 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fa1dc06bdb0 0x7fa1dc06e270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00b7cf0 tx=0x7fa1e4009040 comp rx=0 tx=0).stop 2023-11-02T08:36:11.749 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 10 monclient: shutdown 2023-11-02T08:36:11.749 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:11.749 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f00a8870 msgr2=0x7fa1f00b75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:11.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.744+0000 7fa1f5c5f640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f00a8870 0x7fa1f00b75c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000c530 tx=0x7fa1e000c9f0 comp rx=0 tx=0).stop 2023-11-02T08:36:11.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.745+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 shutdown_connections 2023-11-02T08:36:11.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.745+0000 7fa1f5c5f640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fa1dc06bdb0 0x7fa1dc06e270 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:11.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.745+0000 7fa1f5c5f640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1f0141db0 0x7fa1f01454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:11.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.745+0000 7fa1f5c5f640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1f00b41c0 0x7fa1f01d9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:11.751 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.745+0000 7fa1f5c5f640 1 --2- 172.21.15.151:0/2375962654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1f00a8870 0x7fa1f00b75c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:11.751 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.745+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 >> 172.21.15.151:0/2375962654 conn(0x7fa1f0065e10 msgr2=0x7fa1f013cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:11.751 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.745+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 shutdown_connections 2023-11-02T08:36:11.751 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.745+0000 7fa1f5c5f640 1 -- 172.21.15.151:0/2375962654 wait complete. 2023-11-02T08:36:11.752 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:11.745+0000 7fa1f5c5f640 1 librados: shutdown 2023-11-02T08:36:11.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-02T08:36:13.843 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:36:13.837+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-11-02T08:36:13.955 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:36:13.947+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-11-02T08:36:13.956 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:36:13.949+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-11-02T08:36:14.035 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:36:13.982+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-11-02T08:36:14.466 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:36:14.459+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-11-02T08:36:14.561 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:36:14.548+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-11-02T08:36:14.618 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:36:14.612+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-11-02T08:36:15.160 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:36:15.153+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-11-02T08:36:17.723 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:36:17.715+0000 ac7d640 -1 osd.0 0 waiting for initial osdmap 2023-11-02T08:36:17.728 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:36:17.721+0000 c496640 -1 osd.3 0 waiting for initial osdmap 2023-11-02T08:36:17.747 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:36:17.740+0000 c496640 -1 osd.6 0 waiting for initial osdmap 2023-11-02T08:36:17.775 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:36:17.766+0000 c496640 -1 osd.4 0 waiting for initial osdmap 2023-11-02T08:36:17.777 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:36:17.771+0000 c496640 -1 osd.5 0 waiting for initial osdmap 2023-11-02T08:36:17.812 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:36:17.804+0000 ac7d640 -1 osd.7 0 waiting for initial osdmap 2023-11-02T08:36:17.833 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:36:17.826+0000 ac7d640 -1 osd.2 0 waiting for initial osdmap 2023-11-02T08:36:18.140 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:36:18.120+0000 14265640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:36:18.156 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:36:18.139+0000 14265640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:36:18.191 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:36:18.172+0000 14265640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:36:18.222 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:36:18.205+0000 14265640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:36:18.235 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:36:18.213+0000 14265640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:36:18.283 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:36:18.266+0000 14265640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:36:18.294 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:36:18.277+0000 14265640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:36:18.569 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-02T08:36:18.676 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.673+0000 7f9f2892e640 10 monclient: get_monmap_and_config 2023-11-02T08:36:18.677 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.673+0000 7f9f2892e640 10 monclient: build_initial_monmap 2023-11-02T08:36:18.677 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.673+0000 7f9f2892e640 10 monclient: monmap: 2023-11-02T08:36:18.677 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:36:18.677 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:18.677 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:36:18.674562+0000 2023-11-02T08:36:18.678 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:36:18.674562+0000 2023-11-02T08:36:18.678 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:18.678 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:18.678 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:18.678 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:18.679 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:18.679 INFO:teuthology.misc.health.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:18.679 INFO:teuthology.misc.health.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:18.679 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:18.681 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.674+0000 7f9f2892e640 1 Processor -- start 2023-11-02T08:36:18.682 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.674+0000 7f9f2892e640 1 -- start start 2023-11-02T08:36:18.682 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.674+0000 7f9f2892e640 10 monclient: init 2023-11-02T08:36:18.682 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:18.682 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-11-02T08:36:18.683 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f240b41c0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:18.683 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9f240a8870 0x7f9f241416a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:18.683 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 10 monclient(hunting): picked mon.noname-c con 0x7f9f240a8870 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:18.683 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9f24141be0 0x7f9f2414bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:18.683 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 10 monclient(hunting): picked mon.noname-a con 0x7f9f24141be0 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:18.684 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f24142070 con 0x7f9f240b41c0 2023-11-02T08:36:18.684 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:18.684 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f240a8d00 con 0x7f9f24141be0 2023-11-02T08:36:18.684 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:18.684 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f240b45a0 con 0x7f9f240a8870 2023-11-02T08:36:18.685 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:18.685 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.675+0000 7f9f2892e640 10 monclient(hunting): authenticate will time out at 1350.031366s 2023-11-02T08:36:18.685 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.676+0000 7f9f22d9d640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9f24141be0 0x7f9f2414bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:18.685 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.676+0000 7f9f22d9d640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9f24141be0 0x7f9f2414bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49392/0 (socket says 172.21.15.151:49392) 2023-11-02T08:36:18.685 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.676+0000 7f9f22d9d640 1 -- 172.21.15.151:0/1364731960 learned_addr learned my addr 172.21.15.151:0/1364731960 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:18.685 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.676+0000 7f9f22d9d640 10 monclient(hunting): get_auth_request con 0x7f9f24141be0 auth_method 0 2023-11-02T08:36:18.686 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.676+0000 7f9f22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:18.686 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.676+0000 7f9f22d9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:18.686 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.676+0000 7f9f22d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:18.686 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.677+0000 7f9f22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:18.686 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.677+0000 7f9f22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:18.687 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.677+0000 7f9f22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:18.688 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.679+0000 7f9f22d9d640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2023-11-02T08:36:18.688 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.679+0000 7f9f22d9d640 1 -- 172.21.15.151:0/1364731960 >> v2:172.21.15.178:3300/0 conn(0x7f9f240a8870 msgr2=0x7f9f241416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:18.688 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.679+0000 7f9f22d9d640 1 --2- 172.21.15.151:0/1364731960 >> v2:172.21.15.178:3300/0 conn(0x7f9f240a8870 0x7f9f241416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:18.689 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.679+0000 7f9f22d9d640 1 -- 172.21.15.151:0/1364731960 >> v1:172.21.15.178:6789/0 conn(0x7f9f240b41c0 legacy=0x7f9f2413e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:18.689 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.679+0000 7f9f22d9d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:18.689 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.679+0000 7f9f22d9d640 1 monclient: found mon.noname-a 2023-11-02T08:36:18.689 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.679+0000 7f9f22d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:18.690 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.679+0000 7f9f22d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:18.690 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.679+0000 7f9f22d9d640 1 -- 172.21.15.151:0/1364731960 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f240aa360 con 0x7f9f24141be0 2023-11-02T08:36:18.690 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.681+0000 7f9f22d9d640 1 --2- 172.21.15.151:0/1364731960 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 0x7f9f2414bfb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9f240b2ad0 tx=0x7f9f1c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b49cbbc8697add0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:18.712 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:36:18.704+0000 c496640 -1 osd.1 0 waiting for initial osdmap 2023-11-02T08:36:18.714 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.710+0000 7f9f21d9b640 1 -- 172.21.15.151:0/1364731960 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c02c950 con 0x7f9f24141be0 2023-11-02T08:36:18.714 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:18.714 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:18.715 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 10 monclient: dump: 2023-11-02T08:36:18.715 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:18.715 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:18.715 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:18.716 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:18.716 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:18.716 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:18.716 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:18.716 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:18.717 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:18.717 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:18.717 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 10 monclient: _finish_auth 0 2023-11-02T08:36:18.717 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 10 monclient: _check_auth_tickets 2023-11-02T08:36:18.718 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:18.718 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 1 -- 172.21.15.151:0/1364731960 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1c02cab0 con 0x7f9f24141be0 2023-11-02T08:36:18.718 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f2892e640 5 monclient: authenticate success, global_id 4227 2023-11-02T08:36:18.718 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f2892e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:18.719 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:18.719 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f2892e640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:18.719 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f2892e640 4 set_mon_vals no callback set 2023-11-02T08:36:18.719 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 1 -- 172.21.15.151:0/1364731960 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c02ce40 con 0x7f9f24141be0 2023-11-02T08:36:18.719 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:18.720 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:18.720 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f21d9b640 10 monclient: dump: 2023-11-02T08:36:18.720 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:18.720 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:18.721 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:18.721 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:18.721 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:18.722 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:18.722 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:18.722 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:18.723 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:18.723 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:18.727 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f2892e640 10 monclient: shutdown 2023-11-02T08:36:18.727 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f2892e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:18.728 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f2892e640 1 -- 172.21.15.151:0/1364731960 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 msgr2=0x7f9f2414bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:18.728 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.711+0000 7f9f2892e640 1 --2- 172.21.15.151:0/1364731960 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 0x7f9f2414bfb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9f240b2ad0 tx=0x7f9f1c02bdb0 comp rx=0 tx=0).stop 2023-11-02T08:36:18.728 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.712+0000 7f9f2892e640 1 -- 172.21.15.151:0/1364731960 shutdown_connections 2023-11-02T08:36:18.728 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.712+0000 7f9f2892e640 1 --2- 172.21.15.151:0/1364731960 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 0x7f9f2414bfb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:18.729 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.712+0000 7f9f2892e640 1 --2- 172.21.15.151:0/1364731960 >> v2:172.21.15.178:3300/0 conn(0x7f9f240a8870 0x7f9f241416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:18.729 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.712+0000 7f9f2892e640 1 -- 172.21.15.151:0/1364731960 >> 172.21.15.151:0/1364731960 conn(0x7f9f24065e10 msgr2=0x7f9f24066220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:18.729 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.712+0000 7f9f2892e640 1 -- 172.21.15.151:0/1364731960 shutdown_connections 2023-11-02T08:36:18.729 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.712+0000 7f9f2892e640 1 -- 172.21.15.151:0/1364731960 wait complete. 2023-11-02T08:36:18.730 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.713+0000 7f9f2892e640 10 monclient: build_initial_monmap 2023-11-02T08:36:18.730 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.713+0000 7f9f2892e640 10 monclient: monmap: 2023-11-02T08:36:18.730 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:36:18.730 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:18.730 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:18.731 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2023-11-02T08:36:18.731 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:18.731 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:18.731 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:18.731 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:18.732 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:18.732 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:18.732 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.713+0000 7f9f2892e640 1 librados: starting msgr at 2023-11-02T08:36:18.732 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.713+0000 7f9f2892e640 1 librados: starting objecter 2023-11-02T08:36:18.732 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.713+0000 7f9f2892e640 1 Processor -- start 2023-11-02T08:36:18.733 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 1 -- start start 2023-11-02T08:36:18.733 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 1 librados: setting wanted keys 2023-11-02T08:36:18.733 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 1 librados: calling monclient init 2023-11-02T08:36:18.733 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient: init 2023-11-02T08:36:18.733 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:18.734 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-02T08:36:18.734 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f240a8870 0x7f9f24138cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:18.734 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient(hunting): picked mon.noname-c con 0x7f9f240a8870 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:18.734 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f240b41c0 0x7f9f24143e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:18.735 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient(hunting): picked mon.noname-b con 0x7f9f240b41c0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:18.735 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 0x7f9f240b7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:18.735 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient(hunting): picked mon.noname-a con 0x7f9f24141be0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:18.735 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:18.735 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f241392c0 con 0x7f9f24141be0 2023-11-02T08:36:18.736 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:18.736 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240b7550 con 0x7f9f240b41c0 2023-11-02T08:36:18.736 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:18.736 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f24143720 con 0x7f9f240a8870 2023-11-02T08:36:18.737 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:18.737 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.714+0000 7f9f2892e640 10 monclient(hunting): authenticate will time out at 1350.070553s 2023-11-02T08:36:18.737 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f240a8870 0x7f9f24138cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:18.737 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f22d9d640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 0x7f9f240b7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:18.737 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f22d9d640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 0x7f9f240b7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49402/0 (socket says 172.21.15.151:49402) 2023-11-02T08:36:18.738 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f240a8870 0x7f9f24138cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:37562/0 (socket says 172.21.15.151:37562) 2023-11-02T08:36:18.738 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f22d9d640 1 -- 172.21.15.151:0/3891298924 learned_addr learned my addr 172.21.15.151:0/3891298924 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:18.738 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f22d9d640 10 monclient(hunting): get_auth_request con 0x7f9f24141be0 auth_method 0 2023-11-02T08:36:18.738 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:18.739 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f22d9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:18.739 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f22d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:18.739 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f2259c640 10 monclient(hunting): get_auth_request con 0x7f9f240a8870 auth_method 0 2023-11-02T08:36:18.739 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:18.740 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f2259c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:18.740 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.715+0000 7f9f2259c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:18.740 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.716+0000 7f9f22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:18.741 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.716+0000 7f9f22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:18.741 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.716+0000 7f9f22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:18.741 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.716+0000 7f9f2259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:18.741 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.716+0000 7f9f2259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:18.741 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.716+0000 7f9f2259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:18.742 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2023-11-02T08:36:18.742 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 1 -- 172.21.15.151:0/3891298924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f240a8870 msgr2=0x7f9f24138cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:18.742 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f240a8870 0x7f9f24138cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:18.742 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 1 -- 172.21.15.151:0/3891298924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f240b41c0 msgr2=0x7f9f24143e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:18.743 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f240b41c0 0x7f9f24143e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:18.743 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:18.743 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 1 monclient: found mon.noname-a 2023-11-02T08:36:18.743 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:18.743 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:18.744 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.719+0000 7f9f22d9d640 1 -- 172.21.15.151:0/3891298924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c009b30 con 0x7f9f24141be0 2023-11-02T08:36:18.744 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.722+0000 7f9f22d9d640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 0x7f9f240b7010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c036040 tx=0x7f9f1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:18.744 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.722+0000 7f9f1b7fe640 10 client.?.objecter ms_handle_connect 0x7f9f24141be0 2023-11-02T08:36:18.744 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.722+0000 7f9f1b7fe640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:18.746 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 1 -- 172.21.15.151:0/3891298924 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c03e450 con 0x7f9f24141be0 2023-11-02T08:36:18.746 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:18.746 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:18.747 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 10 monclient: dump: 2023-11-02T08:36:18.747 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:18.747 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:18.747 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:18.747 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:18.748 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:18.748 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:18.748 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:18.748 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:18.749 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:18.749 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:18.749 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 10 monclient: _finish_auth 0 2023-11-02T08:36:18.749 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 10 monclient: _check_auth_tickets 2023-11-02T08:36:18.749 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:18.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 1 -- 172.21.15.151:0/3891298924 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1c03e5d0 con 0x7f9f24141be0 2023-11-02T08:36:18.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:18.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 1 -- 172.21.15.151:0/3891298924 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c03e980 con 0x7f9f24141be0 2023-11-02T08:36:18.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2892e640 5 monclient: authenticate success, global_id 4230 2023-11-02T08:36:18.750 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2892e640 10 monclient: _renew_subs 2023-11-02T08:36:18.751 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2892e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:18.751 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f24143a00 con 0x7f9f24141be0 2023-11-02T08:36:18.751 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:18.751 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:18.751 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f1b7fe640 10 monclient: dump: 2023-11-02T08:36:18.752 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:18.752 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:18.752 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:18.752 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:18.752 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:18.753 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:18.753 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:18.753 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:18.753 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:18.754 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:18.754 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2892e640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:18.754 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2892e640 10 monclient: _renew_subs 2023-11-02T08:36:18.754 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2892e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:18.754 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f24139930 con 0x7f9f24141be0 2023-11-02T08:36:18.755 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2892e640 1 librados: init done 2023-11-02T08:36:18.755 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.727+0000 7f9f2159a640 4 set_mon_vals no callback set 2023-11-02T08:36:18.755 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.728+0000 7f9f2892e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:18.755 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.728+0000 7f9f2892e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:18.755 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.728+0000 7f9f2892e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:18.756 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.728+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f24139470 con 0x7f9f24141be0 2023-11-02T08:36:18.757 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.751+0000 7f9f1b7fe640 1 -- 172.21.15.151:0/3891298924 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 88551+0+0 (secure 0 0 0) 0x7f9f1c03eae0 con 0x7f9f24141be0 2023-11-02T08:36:18.757 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.752+0000 7f9f1b7fe640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f0406bdb0 0x7f9f0406e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:18.757 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.752+0000 7f9f2259c640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f0406bdb0 0x7f9f0406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:18.757 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.752+0000 7f9f2259c640 10 monclient: get_auth_request con 0x7f9f0406bdb0 auth_method 0 2023-11-02T08:36:18.758 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.752+0000 7f9f2259c640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f0406bdb0 0x7f9f0406e270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9f140017f0 tx=0x7f9f14009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:18.758 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:18.752+0000 7f9f1b7fe640 10 client.4230.objecter ms_handle_connect 0x7f9f0406bdb0 2023-11-02T08:36:19.029 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.025+0000 7f9f1b7fe640 1 -- 172.21.15.151:0/3891298924 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2511+0+0 (secure 0 0 0) 0x7f9f1c07b540 con 0x7f9f24141be0 2023-11-02T08:36:19.029 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.025+0000 7f9f1b7fe640 10 client.4230.objecter ms_dispatch 0x7f9f24000b90 osd_map(13..13 src has 1..13) v4 2023-11-02T08:36:19.030 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.025+0000 7f9f1b7fe640 3 client.4230.objecter handle_osd_map got epochs [13,13] > 0 2023-11-02T08:36:19.030 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.025+0000 7f9f1b7fe640 3 client.4230.objecter handle_osd_map decoding full epoch 13 2023-11-02T08:36:19.030 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.025+0000 7f9f1b7fe640 20 client.4230.objecter dump_active .. 0 homeless 2023-11-02T08:36:19.030 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.025+0000 7f9f1b7fe640 1 -- 172.21.15.151:0/3891298924 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f9f1c077180 con 0x7f9f24141be0 2023-11-02T08:36:19.031 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.025+0000 7f9f1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:19.031 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.025+0000 7f9f1b7fe640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:19.154 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:36:19.138+0000 14265640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:36:19.233 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.230+0000 7f9f2892e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:19.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.230+0000 7f9f2892e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:19.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.230+0000 7f9f2892e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:19.234 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.230+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f24067920 con 0x7f9f24141be0 2023-11-02T08:36:19.259 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.253+0000 7f9f1b7fe640 1 -- 172.21.15.151:0/3891298924 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+7506 (secure 0 0 0) 0x7f9f1c07a7a0 con 0x7f9f24141be0 2023-11-02T08:36:19.259 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.253+0000 7f9f1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:19.259 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.253+0000 7f9f1b7fe640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:19.260 INFO:teuthology.misc.health.smithi151.stdout: 2023-11-02T08:36:19.260 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":13,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:36:18.326834+0000","last_up_change":"2023-11-02T08:36:18.326834+0000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":86098},{"type":"v1","addr":"172.21.15.151:6809","nonce":86098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":86098},{"type":"v1","addr":"172.21.15.151:6811","nonce":86098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":86098},{"type":"v1","addr":"172.21.15.151:6815","nonce":86098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":86098},{"type":"v1","addr":"172.21.15.151:6813","nonce":86098}]},"public_addr":"172.21.15.151:6809/86098","cluster_addr":"172.21.15.151:6811/86098","heartbeat_back_addr":"172.21.15.151:6815/86098","heartbeat_front_addr":"172.21.15.151:6813/86098","state":["exists","up"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:36:19.261 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 10 librados: watch_flush enter 2023-11-02T08:36:19.261 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 10 librados: watch_flush exit 2023-11-02T08:36:19.262 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 20 client.4230.objecter shutdown clearing up homeless session... 2023-11-02T08:36:19.262 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 10 client.4230.objecter successfully canceled tick 2023-11-02T08:36:19.262 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f0406bdb0 msgr2=0x7f9f0406e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:19.262 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f0406bdb0 0x7f9f0406e270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9f140017f0 tx=0x7f9f14009040 comp rx=0 tx=0).stop 2023-11-02T08:36:19.262 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 10 monclient: shutdown 2023-11-02T08:36:19.263 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:19.263 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 msgr2=0x7f9f240b7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:19.263 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 0x7f9f240b7010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c036040 tx=0x7f9f1c004a40 comp rx=0 tx=0).stop 2023-11-02T08:36:19.263 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 shutdown_connections 2023-11-02T08:36:19.263 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f24141be0 0x7f9f240b7010 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:19.264 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f0406bdb0 0x7f9f0406e270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:19.264 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.255+0000 7f9f2892e640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f240b41c0 0x7f9f24143e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:19.264 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.256+0000 7f9f2892e640 1 --2- 172.21.15.151:0/3891298924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f240a8870 0x7f9f24138cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:19.265 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.256+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 >> 172.21.15.151:0/3891298924 conn(0x7f9f24065e10 msgr2=0x7f9f2413cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:19.265 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.256+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 shutdown_connections 2023-11-02T08:36:19.265 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.256+0000 7f9f2892e640 1 -- 172.21.15.151:0/3891298924 wait complete. 2023-11-02T08:36:19.266 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:19.256+0000 7f9f2892e640 1 librados: shutdown 2023-11-02T08:36:19.269 DEBUG:teuthology.misc:1 of 8 OSDs are up 2023-11-02T08:36:26.076 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-02T08:36:26.168 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.165+0000 7fe9dbf5c640 10 monclient: get_monmap_and_config 2023-11-02T08:36:26.169 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.165+0000 7fe9dbf5c640 10 monclient: build_initial_monmap 2023-11-02T08:36:26.169 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.165+0000 7fe9dbf5c640 10 monclient: monmap: 2023-11-02T08:36:26.169 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:36:26.169 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.170 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:36:26.166650+0000 2023-11-02T08:36:26.170 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:36:26.166650+0000 2023-11-02T08:36:26.170 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:26.170 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.170 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:26.171 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:26.171 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:26.171 INFO:teuthology.misc.health.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:26.171 INFO:teuthology.misc.health.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:26.171 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:26.174 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 1 Processor -- start 2023-11-02T08:36:26.174 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 1 -- start start 2023-11-02T08:36:26.175 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 10 monclient: init 2023-11-02T08:36:26.175 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:26.175 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-11-02T08:36:26.175 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe9d40b41c0 0x7fe9d40a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.175 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 10 monclient(hunting): picked mon.noname-c con 0x7fe9d40b41c0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:26.176 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9d413e780 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:26.176 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d4141db0 0x7fe9d413ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.176 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.166+0000 7fe9dbf5c640 10 monclient(hunting): picked mon.noname-b con 0x7fe9d4141db0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:26.176 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9dbf5c640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d40aa360 con 0x7fe9d413e780 2023-11-02T08:36:26.176 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9dbf5c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.177 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9dbf5c640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d40aa160 con 0x7fe9d40b41c0 2023-11-02T08:36:26.177 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9dbf5c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.177 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9dbf5c640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40b45a0 con 0x7fe9d4141db0 2023-11-02T08:36:26.177 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9dbf5c640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:26.177 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9dbf5c640 10 monclient(hunting): authenticate will time out at 1357.522900s 2023-11-02T08:36:26.178 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9da4f9640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d4141db0 0x7fe9d413ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.178 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9d9cf8640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe9d40b41c0 0x7fe9d40a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.178 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9da4f9640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d4141db0 0x7fe9d413ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:55044/0 (socket says 172.21.15.151:55044) 2023-11-02T08:36:26.178 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9da4f9640 1 -- 172.21.15.151:0/427779819 learned_addr learned my addr 172.21.15.151:0/427779819 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:26.178 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9da4f9640 10 monclient(hunting): get_auth_request con 0x7fe9d4141db0 auth_method 0 2023-11-02T08:36:26.179 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9da4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.179 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9da4f9640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.179 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.167+0000 7fe9da4f9640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.179 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.168+0000 7fe9d9cf8640 10 monclient(hunting): get_auth_request con 0x7fe9d40b41c0 auth_method 0 2023-11-02T08:36:26.179 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.168+0000 7fe9d9cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.180 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.168+0000 7fe9d9cf8640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.180 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.168+0000 7fe9d9cf8640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.180 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.168+0000 7fe9da4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.180 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.168+0000 7fe9da4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.180 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.168+0000 7fe9da4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.180 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.169+0000 7fe9d9cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.181 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.169+0000 7fe9d9cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.181 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.169+0000 7fe9d9cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.181 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.170+0000 7fe9da4f9640 10 monclient(hunting): handle_auth_done global_id 4208 payload 274 2023-11-02T08:36:26.181 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.170+0000 7fe9da4f9640 1 -- 172.21.15.151:0/427779819 >> v2:172.21.15.178:3300/0 conn(0x7fe9d40b41c0 msgr2=0x7fe9d40a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:26.181 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.170+0000 7fe9da4f9640 1 --2- 172.21.15.151:0/427779819 >> v2:172.21.15.178:3300/0 conn(0x7fe9d40b41c0 0x7fe9d40a8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.182 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.170+0000 7fe9da4f9640 1 -- 172.21.15.151:0/427779819 >> v1:172.21.15.178:6789/0 conn(0x7fe9d413e780 legacy=0x7fe9d41416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.182 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.170+0000 7fe9da4f9640 10 monclient: _finish_hunting 0 2023-11-02T08:36:26.182 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.170+0000 7fe9da4f9640 1 monclient: found mon.noname-b 2023-11-02T08:36:26.182 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.170+0000 7fe9da4f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:26.182 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.170+0000 7fe9da4f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:26.183 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.170+0000 7fe9da4f9640 1 -- 172.21.15.151:0/427779819 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d40aa640 con 0x7fe9d4141db0 2023-11-02T08:36:26.187 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.172+0000 7fe9da4f9640 1 --2- 172.21.15.151:0/427779819 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d4141db0 0x7fe9d413ebf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc009e50 tx=0x7fe9cc02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cab802ffc2bcfcc8 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:26.187 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 1 -- 172.21.15.151:0/427779819 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc02c9e0 con 0x7fe9d4141db0 2023-11-02T08:36:26.187 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:26.188 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:26.188 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 10 monclient: dump: 2023-11-02T08:36:26.188 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:26.188 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.188 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.189 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.189 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:26.189 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.189 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:26.189 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:26.190 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:26.190 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:26.190 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 10 monclient: _finish_auth 0 2023-11-02T08:36:26.190 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 10 monclient: _check_auth_tickets 2023-11-02T08:36:26.191 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:26.191 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 1 -- 172.21.15.151:0/427779819 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc02cb40 con 0x7fe9d4141db0 2023-11-02T08:36:26.191 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9dbf5c640 5 monclient: authenticate success, global_id 4208 2023-11-02T08:36:26.191 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9dbf5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:26.191 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:26.191 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9dbf5c640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:26.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9dbf5c640 4 set_mon_vals no callback set 2023-11-02T08:36:26.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 1 -- 172.21.15.151:0/427779819 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0343f0 con 0x7fe9d4141db0 2023-11-02T08:36:26.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:26.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:26.192 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.175+0000 7fe9d8cf6640 10 monclient: dump: 2023-11-02T08:36:26.193 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:26.193 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.193 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.193 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.193 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:26.194 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.194 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:26.194 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:26.194 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:26.194 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:26.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 10 monclient: shutdown 2023-11-02T08:36:26.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:26.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/427779819 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d4141db0 msgr2=0x7fe9d413ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 1 --2- 172.21.15.151:0/427779819 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d4141db0 0x7fe9d413ebf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc009e50 tx=0x7fe9cc02be40 comp rx=0 tx=0).stop 2023-11-02T08:36:26.195 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/427779819 shutdown_connections 2023-11-02T08:36:26.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 1 --2- 172.21.15.151:0/427779819 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d4141db0 0x7fe9d413ebf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 1 --2- 172.21.15.151:0/427779819 >> v2:172.21.15.178:3300/0 conn(0x7fe9d40b41c0 0x7fe9d40a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/427779819 >> 172.21.15.151:0/427779819 conn(0x7fe9d4065e10 msgr2=0x7fe9d4066220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:26.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/427779819 shutdown_connections 2023-11-02T08:36:26.196 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.176+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/427779819 wait complete. 2023-11-02T08:36:26.197 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.177+0000 7fe9dbf5c640 10 monclient: build_initial_monmap 2023-11-02T08:36:26.197 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.177+0000 7fe9dbf5c640 10 monclient: monmap: 2023-11-02T08:36:26.197 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2023-11-02T08:36:26.197 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:26.197 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:26.197 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2023-11-02T08:36:26.198 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:26.198 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.198 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:26.198 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:26.198 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:26.199 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:26.199 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.177+0000 7fe9dbf5c640 1 librados: starting msgr at 2023-11-02T08:36:26.199 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.177+0000 7fe9dbf5c640 1 librados: starting objecter 2023-11-02T08:36:26.199 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.177+0000 7fe9dbf5c640 1 Processor -- start 2023-11-02T08:36:26.199 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.177+0000 7fe9dbf5c640 1 -- start start 2023-11-02T08:36:26.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.177+0000 7fe9dbf5c640 1 librados: setting wanted keys 2023-11-02T08:36:26.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.177+0000 7fe9dbf5c640 1 librados: calling monclient init 2023-11-02T08:36:26.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.177+0000 7fe9dbf5c640 10 monclient: init 2023-11-02T08:36:26.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:26.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-02T08:36:26.200 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9d40b41c0 0x7fe9d41438f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.201 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient(hunting): picked mon.noname-b con 0x7fe9d40b41c0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:26.201 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d413e780 0x7fe9d4135280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.201 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient(hunting): picked mon.noname-c con 0x7fe9d413e780 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:26.201 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9d4141db0 0x7fe9d41357c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.201 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient(hunting): picked mon.noname-a con 0x7fe9d4141db0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:26.202 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.202 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d4135d00 con 0x7fe9d4141db0 2023-11-02T08:36:26.202 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.202 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d4144370 con 0x7fe9d40b41c0 2023-11-02T08:36:26.202 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.203 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d4145390 con 0x7fe9d413e780 2023-11-02T08:36:26.203 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:26.203 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9dbf5c640 10 monclient(hunting): authenticate will time out at 1357.534072s 2023-11-02T08:36:26.203 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9d94f7640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d413e780 0x7fe9d4135280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.203 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9da4f9640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9d4141db0 0x7fe9d41357c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.203 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.178+0000 7fe9d9cf8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9d40b41c0 0x7fe9d41438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.204 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d94f7640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d413e780 0x7fe9d4135280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:55054/0 (socket says 172.21.15.151:55054) 2023-11-02T08:36:26.204 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d9cf8640 10 monclient(hunting): get_auth_request con 0x7fe9d40b41c0 auth_method 0 2023-11-02T08:36:26.204 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d94f7640 1 -- 172.21.15.151:0/972417865 learned_addr learned my addr 172.21.15.151:0/972417865 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:26.204 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d9cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.204 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d9cf8640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.205 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d9cf8640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.205 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d94f7640 10 monclient(hunting): get_auth_request con 0x7fe9d413e780 auth_method 0 2023-11-02T08:36:26.205 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d94f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.205 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d94f7640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.205 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d94f7640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.206 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9da4f9640 10 monclient(hunting): get_auth_request con 0x7fe9d4141db0 auth_method 0 2023-11-02T08:36:26.206 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9da4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.206 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9da4f9640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.206 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9da4f9640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.206 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d9cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.206 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d9cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.207 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d9cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.207 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d94f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.207 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d94f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.207 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9d94f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.207 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9da4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9da4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.179+0000 7fe9da4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2023-11-02T08:36:26.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 1 -- 172.21.15.151:0/972417865 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d413e780 msgr2=0x7fe9d4135280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.208 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d413e780 0x7fe9d4135280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 1 -- 172.21.15.151:0/972417865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9d4141db0 msgr2=0x7fe9d41357c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9d4141db0 0x7fe9d41357c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 10 monclient: _finish_hunting 0 2023-11-02T08:36:26.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 1 monclient: found mon.noname-b 2023-11-02T08:36:26.209 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:26.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:26.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.182+0000 7fe9d9cf8640 1 -- 172.21.15.151:0/972417865 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc009b30 con 0x7fe9d40b41c0 2023-11-02T08:36:26.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.184+0000 7fe9d9cf8640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9d40b41c0 0x7fe9d41438f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000e850 tx=0x7fe9d000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:26.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.184+0000 7fe9ca7fc640 10 client.?.objecter ms_handle_connect 0x7fe9d40b41c0 2023-11-02T08:36:26.210 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.184+0000 7fe9ca7fc640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:26.212 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 1 -- 172.21.15.151:0/972417865 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00095b0 con 0x7fe9d40b41c0 2023-11-02T08:36:26.212 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:26.212 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:26.212 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 10 monclient: dump: 2023-11-02T08:36:26.213 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:26.213 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.213 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.213 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.213 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:26.214 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.214 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:26.214 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:26.214 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:26.214 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:26.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:26.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:26.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:26.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 1 -- 172.21.15.151:0/972417865 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d000c940 con 0x7fe9d40b41c0 2023-11-02T08:36:26.215 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:26.216 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 1 -- 172.21.15.151:0/972417865 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000ccd0 con 0x7fe9d40b41c0 2023-11-02T08:36:26.216 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9dbf5c640 5 monclient: authenticate success, global_id 4273 2023-11-02T08:36:26.216 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9dbf5c640 10 monclient: _renew_subs 2023-11-02T08:36:26.216 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9dbf5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:26.216 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d4145670 con 0x7fe9d40b41c0 2023-11-02T08:36:26.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:26.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:26.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9cbfff640 4 set_mon_vals no callback set 2023-11-02T08:36:26.217 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9ca7fc640 10 monclient: dump: 2023-11-02T08:36:26.217 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2023-11-02T08:36:26.217 INFO:teuthology.misc.health.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.218 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.218 INFO:teuthology.misc.health.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.218 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:26.218 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.218 INFO:teuthology.misc.health.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:26.219 INFO:teuthology.misc.health.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:26.219 INFO:teuthology.misc.health.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:26.219 INFO:teuthology.misc.health.smithi151.stderr: 2023-11-02T08:36:26.219 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9dbf5c640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:26.219 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9dbf5c640 10 monclient: _renew_subs 2023-11-02T08:36:26.219 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9dbf5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:26.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d4143e30 con 0x7fe9d40b41c0 2023-11-02T08:36:26.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.188+0000 7fe9dbf5c640 1 librados: init done 2023-11-02T08:36:26.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.189+0000 7fe9dbf5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:26.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.189+0000 7fe9dbf5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:26.220 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.189+0000 7fe9dbf5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:26.221 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.189+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d4067920 con 0x7fe9d40b41c0 2023-11-02T08:36:26.222 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.212+0000 7fe9ca7fc640 1 -- 172.21.15.151:0/972417865 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 4) v1 ==== 88551+0+0 (secure 0 0 0) 0x7fe9d0026480 con 0x7fe9d40b41c0 2023-11-02T08:36:26.222 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.212+0000 7fe9ca7fc640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe9b806bd80 0x7fe9b806e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.222 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.212+0000 7fe9d94f7640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe9b806bd80 0x7fe9b806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.222 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.213+0000 7fe9d94f7640 10 monclient: get_auth_request con 0x7fe9b806bd80 auth_method 0 2023-11-02T08:36:26.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.213+0000 7fe9d94f7640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe9b806bd80 0x7fe9b806e240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9d413dd80 tx=0x7fe9c4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:26.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.213+0000 7fe9ca7fc640 10 client.4273.objecter ms_handle_connect 0x7fe9b806bd80 2023-11-02T08:36:26.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.214+0000 7fe9ca7fc640 1 -- 172.21.15.151:0/972417865 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe9d000ce30 con 0x7fe9d40b41c0 2023-11-02T08:36:26.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.214+0000 7fe9ca7fc640 10 client.4273.objecter ms_dispatch 0x7fe9d4000b90 osd_map(18..18 src has 1..18) v4 2023-11-02T08:36:26.223 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.214+0000 7fe9ca7fc640 3 client.4273.objecter handle_osd_map got epochs [18,18] > 0 2023-11-02T08:36:26.224 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.214+0000 7fe9ca7fc640 3 client.4273.objecter handle_osd_map decoding full epoch 18 2023-11-02T08:36:26.224 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.215+0000 7fe9ca7fc640 20 client.4273.objecter dump_active .. 0 homeless 2023-11-02T08:36:26.493 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.489+0000 7fe9ca7fc640 1 -- 172.21.15.151:0/972417865 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fe9d00548e0 con 0x7fe9d40b41c0 2023-11-02T08:36:26.493 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.489+0000 7fe9ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:26.493 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.489+0000 7fe9ca7fc640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:26.699 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.696+0000 7fe9dbf5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:26.700 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.696+0000 7fe9dbf5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:26.700 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.696+0000 7fe9dbf5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:26.700 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.696+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9d413da80 con 0x7fe9d40b41c0 2023-11-02T08:36:26.793 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.790+0000 7fe9ca7fc640 1 -- 172.21.15.151:0/972417865 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12430 (secure 0 0 0) 0x7fe9d0021090 con 0x7fe9d40b41c0 2023-11-02T08:36:26.794 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.790+0000 7fe9ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:26.794 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.790+0000 7fe9ca7fc640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:26.794 INFO:teuthology.misc.health.smithi151.stdout: 2023-11-02T08:36:26.794 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":18,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:36:25.209464+0000","last_up_change":"2023-11-02T08:36:19.358160+0000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-02T08:36:21.174515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":86098},{"type":"v1","addr":"172.21.15.151:6809","nonce":86098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":86098},{"type":"v1","addr":"172.21.15.151:6811","nonce":86098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":86098},{"type":"v1","addr":"172.21.15.151:6815","nonce":86098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":86098},{"type":"v1","addr":"172.21.15.151:6813","nonce":86098}]},"public_addr":"172.21.15.151:6809/86098","cluster_addr":"172.21.15.151:6811/86098","heartbeat_back_addr":"172.21.15.151:6815/86098","heartbeat_front_addr":"172.21.15.151:6813/86098","state":["exists","up"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":86101},{"type":"v1","addr":"172.21.15.151:6801","nonce":86101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":86101},{"type":"v1","addr":"172.21.15.151:6803","nonce":86101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":86101},{"type":"v1","addr":"172.21.15.151:6807","nonce":86101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":86101},{"type":"v1","addr":"172.21.15.151:6805","nonce":86101}]},"public_addr":"172.21.15.151:6801/86101","cluster_addr":"172.21.15.151:6803/86101","heartbeat_back_addr":"172.21.15.151:6807/86101","heartbeat_front_addr":"172.21.15.151:6805/86101","state":["exists","up"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":86100},{"type":"v1","addr":"172.21.15.151:6817","nonce":86100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":86100},{"type":"v1","addr":"172.21.15.151:6819","nonce":86100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":86100},{"type":"v1","addr":"172.21.15.151:6823","nonce":86100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":86100},{"type":"v1","addr":"172.21.15.151:6821","nonce":86100}]},"public_addr":"172.21.15.151:6817/86100","cluster_addr":"172.21.15.151:6819/86100","heartbeat_back_addr":"172.21.15.151:6823/86100","heartbeat_front_addr":"172.21.15.151:6821/86100","state":["exists","up"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":86099},{"type":"v1","addr":"172.21.15.151:6825","nonce":86099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":86099},{"type":"v1","addr":"172.21.15.151:6827","nonce":86099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":86099},{"type":"v1","addr":"172.21.15.151:6831","nonce":86099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":86099},{"type":"v1","addr":"172.21.15.151:6829","nonce":86099}]},"public_addr":"172.21.15.151:6825/86099","cluster_addr":"172.21.15.151:6827/86099","heartbeat_back_addr":"172.21.15.151:6831/86099","heartbeat_front_addr":"172.21.15.151:6829/86099","state":["exists","up"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":86120},{"type":"v1","addr":"172.21.15.178:6811","nonce":86120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":86120},{"type":"v1","addr":"172.21.15.178:6813","nonce":86120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":86120},{"type":"v1","addr":"172.21.15.178:6817","nonce":86120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":86120},{"type":"v1","addr":"172.21.15.178:6815","nonce":86120}]},"public_addr":"172.21.15.178:6811/86120","cluster_addr":"172.21.15.178:6813/86120","heartbeat_back_addr":"172.21.15.178:6817/86120","heartbeat_front_addr":"172.21.15.178:6815/86120","state":["exists","up"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":86122},{"type":"v1","addr":"172.21.15.178:6819","nonce":86122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":86122},{"type":"v1","addr":"172.21.15.178:6821","nonce":86122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":86122},{"type":"v1","addr":"172.21.15.178:6825","nonce":86122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":86122},{"type":"v1","addr":"172.21.15.178:6823","nonce":86122}]},"public_addr":"172.21.15.178:6819/86122","cluster_addr":"172.21.15.178:6821/86122","heartbeat_back_addr":"172.21.15.178:6825/86122","heartbeat_front_addr":"172.21.15.178:6823/86122","state":["exists","up"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":86121},{"type":"v1","addr":"172.21.15.178:6827","nonce":86121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":86121},{"type":"v1","addr":"172.21.15.178:6829","nonce":86121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":86121},{"type":"v1","addr":"172.21.15.178:6833","nonce":86121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":86121},{"type":"v1","addr":"172.21.15.178:6831","nonce":86121}]},"public_addr":"172.21.15.178:6827/86121","cluster_addr":"172.21.15.178:6829/86121","heartbeat_back_addr":"172.21.15.178:6833/86121","heartbeat_front_addr":"172.21.15.178:6831/86121","state":["exists","up"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":86123},{"type":"v1","addr":"172.21.15.178:6803","nonce":86123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":86123},{"type":"v1","addr":"172.21.15.178:6805","nonce":86123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":86123},{"type":"v1","addr":"172.21.15.178:6809","nonce":86123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":86123},{"type":"v1","addr":"172.21.15.178:6807","nonce":86123}]},"public_addr":"172.21.15.178:6803/86123","cluster_addr":"172.21.15.178:6805/86123","heartbeat_back_addr":"172.21.15.178:6809/86123","heartbeat_front_addr":"172.21.15.178:6807/86123","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.010757+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:17.073234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.605027+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.579731+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.490219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:15.935790+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:15.992196+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.123838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:36:26.796 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 10 librados: watch_flush enter 2023-11-02T08:36:26.796 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 10 librados: watch_flush exit 2023-11-02T08:36:26.796 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 20 client.4273.objecter shutdown clearing up homeless session... 2023-11-02T08:36:26.797 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 10 client.4273.objecter successfully canceled tick 2023-11-02T08:36:26.797 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe9b806bd80 msgr2=0x7fe9b806e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.797 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe9b806bd80 0x7fe9b806e240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9d413dd80 tx=0x7fe9c4009040 comp rx=0 tx=0).stop 2023-11-02T08:36:26.797 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 10 monclient: shutdown 2023-11-02T08:36:26.797 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:26.798 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9d40b41c0 msgr2=0x7fe9d41438f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.798 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9d40b41c0 0x7fe9d41438f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000e850 tx=0x7fe9d000ed10 comp rx=0 tx=0).stop 2023-11-02T08:36:26.798 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 shutdown_connections 2023-11-02T08:36:26.798 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9d4141db0 0x7fe9d41357c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.798 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9d413e780 0x7fe9d4135280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.799 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fe9b806bd80 0x7fe9b806e240 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.799 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 --2- 172.21.15.151:0/972417865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9d40b41c0 0x7fe9d41438f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.799 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 >> 172.21.15.151:0/972417865 conn(0x7fe9d4065e10 msgr2=0x7fe9d413a720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:26.799 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 shutdown_connections 2023-11-02T08:36:26.799 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 -- 172.21.15.151:0/972417865 wait complete. 2023-11-02T08:36:26.799 INFO:teuthology.misc.health.smithi151.stderr:2023-11-02T08:36:26.792+0000 7fe9dbf5c640 1 librados: shutdown 2023-11-02T08:36:26.806 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-02T08:36:26.806 INFO:tasks.ceph:Creating RBD pool 2023-11-02T08:36:26.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-02T08:36:26.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.903+0000 7f3a634a7640 10 monclient: get_monmap_and_config 2023-11-02T08:36:26.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.903+0000 7f3a634a7640 10 monclient: build_initial_monmap 2023-11-02T08:36:26.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.903+0000 7f3a634a7640 10 monclient: monmap: 2023-11-02T08:36:26.907 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:26.908 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.908 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:26.904860+0000 2023-11-02T08:36:26.908 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:26.904860+0000 2023-11-02T08:36:26.908 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:26.908 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.908 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:26.909 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:26.909 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:26.909 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:26.909 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:26.909 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:26.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.904+0000 7f3a634a7640 1 Processor -- start 2023-11-02T08:36:26.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.904+0000 7f3a634a7640 1 -- start start 2023-11-02T08:36:26.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.904+0000 7f3a634a7640 10 monclient: init 2023-11-02T08:36:26.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:26.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-11-02T08:36:26.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3a5c0b7be0 0x7f3a5c0a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 10 monclient(hunting): picked mon.noname-a con 0x7f3a5c0b7be0 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:26.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a5c0a9420 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:26.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3a5c13e930 0x7f3a5c13eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 10 monclient(hunting): picked mon.noname-c con 0x7f3a5c13e930 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:26.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5c0aa260 con 0x7f3a5c0a9420 2023-11-02T08:36:26.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5c0b7fc0 con 0x7f3a5c0b7be0 2023-11-02T08:36:26.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5c0aa4b0 con 0x7f3a5c13e930 2023-11-02T08:36:26.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:26.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a634a7640 10 monclient(hunting): authenticate will time out at 1358.261168s 2023-11-02T08:36:26.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.905+0000 7f3a61243640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3a5c0b7be0 0x7f3a5c0a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61a44640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3a5c13e930 0x7f3a5c13eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61243640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3a5c0b7be0 0x7f3a5c0a8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47870/0 (socket says 172.21.15.151:47870) 2023-11-02T08:36:26.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61243640 1 -- 172.21.15.151:0/924035338 learned_addr learned my addr 172.21.15.151:0/924035338 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:26.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61243640 10 monclient(hunting): get_auth_request con 0x7f3a5c0b7be0 auth_method 0 2023-11-02T08:36:26.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61243640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61243640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61243640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61243640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61243640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61243640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61a44640 10 monclient(hunting): get_auth_request con 0x7f3a5c13e930 auth_method 0 2023-11-02T08:36:26.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61a44640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.906+0000 7f3a61a44640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.907+0000 7f3a61a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.907+0000 7f3a61a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.907+0000 7f3a61a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.908+0000 7f3a61243640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2023-11-02T08:36:26.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.908+0000 7f3a61243640 1 -- 172.21.15.151:0/924035338 >> v2:172.21.15.178:3300/0 conn(0x7f3a5c13e930 msgr2=0x7f3a5c13eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:26.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.908+0000 7f3a61243640 1 --2- 172.21.15.151:0/924035338 >> v2:172.21.15.178:3300/0 conn(0x7f3a5c13e930 0x7f3a5c13eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.908+0000 7f3a61243640 1 -- 172.21.15.151:0/924035338 >> v1:172.21.15.178:6789/0 conn(0x7f3a5c0a9420 legacy=0x7f3a5c144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.908+0000 7f3a61243640 10 monclient: _finish_hunting 0 2023-11-02T08:36:26.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.908+0000 7f3a61243640 1 monclient: found mon.noname-a 2023-11-02T08:36:26.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.908+0000 7f3a61243640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:26.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.908+0000 7f3a61243640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:26.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.908+0000 7f3a61243640 1 -- 172.21.15.151:0/924035338 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c0aa790 con 0x7f3a5c0b7be0 2023-11-02T08:36:26.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.910+0000 7f3a61243640 1 --2- 172.21.15.151:0/924035338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c0b7be0 0x7f3a5c0a8870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c06be70 tx=0x7f3a5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e607f1a0923ac4 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:26.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.913+0000 7f3a53fff640 1 -- 172.21.15.151:0/924035338 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58030860 con 0x7f3a5c0b7be0 2023-11-02T08:36:26.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.913+0000 7f3a53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:26.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.913+0000 7f3a53fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:26.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 10 monclient: dump: 2023-11-02T08:36:26.938 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:26.938 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.939 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.939 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.939 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:26.940 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.940 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:26.940 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:26.940 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:26.940 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:26.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 10 monclient: _finish_auth 0 2023-11-02T08:36:26.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:26.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:26.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 1 -- 172.21.15.151:0/924035338 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a580309c0 con 0x7f3a5c0b7be0 2023-11-02T08:36:26.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 5 monclient: authenticate success, global_id 4242 2023-11-02T08:36:26.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:26.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:26.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:26.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 4 set_mon_vals no callback set 2023-11-02T08:36:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 1 -- 172.21.15.151:0/924035338 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58030d50 con 0x7f3a5c0b7be0 2023-11-02T08:36:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:26.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a53fff640 10 monclient: dump: 2023-11-02T08:36:26.945 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:26.945 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.946 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.947 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:26.947 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.947 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:26.948 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:26.948 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:26.948 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:26.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 10 monclient: shutdown 2023-11-02T08:36:26.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:26.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 1 -- 172.21.15.151:0/924035338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c0b7be0 msgr2=0x7f3a5c0a8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 1 --2- 172.21.15.151:0/924035338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c0b7be0 0x7f3a5c0a8870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c06be70 tx=0x7f3a5802b9a0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 1 -- 172.21.15.151:0/924035338 shutdown_connections 2023-11-02T08:36:26.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 1 --2- 172.21.15.151:0/924035338 >> v2:172.21.15.178:3300/0 conn(0x7f3a5c13e930 0x7f3a5c13eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 1 --2- 172.21.15.151:0/924035338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c0b7be0 0x7f3a5c0a8870 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 1 -- 172.21.15.151:0/924035338 >> 172.21.15.151:0/924035338 conn(0x7f3a5c06c510 msgr2=0x7f3a5c0b7260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:26.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 1 -- 172.21.15.151:0/924035338 shutdown_connections 2023-11-02T08:36:26.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.914+0000 7f3a634a7640 1 -- 172.21.15.151:0/924035338 wait complete. 2023-11-02T08:36:26.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.915+0000 7f3a634a7640 10 monclient: build_initial_monmap 2023-11-02T08:36:26.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.915+0000 7f3a634a7640 10 monclient: monmap: 2023-11-02T08:36:26.956 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:26.957 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:26.957 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:26.957 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:26.957 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:26.958 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.958 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:26.959 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:26.959 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:26.959 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:26.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.915+0000 7f3a634a7640 1 librados: starting msgr at 2023-11-02T08:36:26.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.915+0000 7f3a634a7640 1 librados: starting objecter 2023-11-02T08:36:26.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.915+0000 7f3a634a7640 1 Processor -- start 2023-11-02T08:36:26.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 1 -- start start 2023-11-02T08:36:26.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 1 librados: setting wanted keys 2023-11-02T08:36:26.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 1 librados: calling monclient init 2023-11-02T08:36:26.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient: init 2023-11-02T08:36:26.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:26.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-02T08:36:26.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c0a9420 0x7f3a5c1d67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient(hunting): picked mon.noname-b con 0x7f3a5c0a9420 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:26.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a5c0b7be0 0x7f3a5c0ab260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient(hunting): picked mon.noname-c con 0x7f3a5c0b7be0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:26.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c13e930 0x7f3a5c0abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient(hunting): picked mon.noname-a con 0x7f3a5c13e930 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:26.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0abb50 con 0x7f3a5c13e930 2023-11-02T08:36:26.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0ab7a0 con 0x7f3a5c0a9420 2023-11-02T08:36:26.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:26.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0ac520 con 0x7f3a5c0b7be0 2023-11-02T08:36:26.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:26.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.916+0000 7f3a634a7640 10 monclient(hunting): authenticate will time out at 1358.272429s 2023-11-02T08:36:26.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c13e930 0x7f3a5c0abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61243640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c0a9420 0x7f3a5c1d67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a60a42640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a5c0b7be0 0x7f3a5c0ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c13e930 0x7f3a5c0abfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47878/0 (socket says 172.21.15.151:47878) 2023-11-02T08:36:26.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 1 -- 172.21.15.151:0/2050861053 learned_addr learned my addr 172.21.15.151:0/2050861053 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:26.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 10 monclient(hunting): get_auth_request con 0x7f3a5c13e930 auth_method 0 2023-11-02T08:36:26.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a60a42640 10 monclient(hunting): get_auth_request con 0x7f3a5c0b7be0 auth_method 0 2023-11-02T08:36:26.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a60a42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a60a42640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.968 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a60a42640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61243640 10 monclient(hunting): get_auth_request con 0x7f3a5c0a9420 auth_method 0 2023-11-02T08:36:26.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61243640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:26.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61243640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:26.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61243640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:26.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.917+0000 7f3a61a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.918+0000 7f3a60a42640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.918+0000 7f3a60a42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.918+0000 7f3a60a42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.918+0000 7f3a61243640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:26.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.918+0000 7f3a61243640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:26.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.918+0000 7f3a61243640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:26.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2023-11-02T08:36:26.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 1 -- 172.21.15.151:0/2050861053 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a5c0b7be0 msgr2=0x7f3a5c0ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a5c0b7be0 0x7f3a5c0ab260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 1 -- 172.21.15.151:0/2050861053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c0a9420 msgr2=0x7f3a5c1d67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:26.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c0a9420 0x7f3a5c1d67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:26.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 10 monclient: _finish_hunting 0 2023-11-02T08:36:26.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 1 monclient: found mon.noname-a 2023-11-02T08:36:26.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:26.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:26.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.920+0000 7f3a61a44640 1 -- 172.21.15.151:0/2050861053 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a580094d0 con 0x7f3a5c13e930 2023-11-02T08:36:26.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.923+0000 7f3a61a44640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c13e930 0x7f3a5c0abfe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400ec10 tx=0x7f3a5400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:26.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.923+0000 7f3a51ffb640 10 client.?.objecter ms_handle_connect 0x7f3a5c13e930 2023-11-02T08:36:26.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.923+0000 7f3a51ffb640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:26.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 1 -- 172.21.15.151:0/2050861053 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5400cd20 con 0x7f3a5c13e930 2023-11-02T08:36:26.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:26.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:26.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 10 monclient: dump: 2023-11-02T08:36:26.975 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:26.975 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.976 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.976 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.976 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:26.977 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.977 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:26.977 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:26.977 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:26.977 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:26.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 10 monclient: _finish_auth 0 2023-11-02T08:36:26.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 10 monclient: _check_auth_tickets 2023-11-02T08:36:26.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:26.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 1 -- 172.21.15.151:0/2050861053 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5400f040 con 0x7f3a5c13e930 2023-11-02T08:36:26.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:26.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 1 -- 172.21.15.151:0/2050861053 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540146c0 con 0x7f3a5c13e930 2023-11-02T08:36:26.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a634a7640 5 monclient: authenticate success, global_id 4245 2023-11-02T08:36:26.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:26.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:26.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a51ffb640 10 monclient: dump: 2023-11-02T08:36:26.981 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:26.981 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:26.982 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.982 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:26.982 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:26.982 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:26.982 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:26.982 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:26.983 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:26.983 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:26.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a634a7640 10 monclient: _renew_subs 2023-11-02T08:36:26.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a634a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:26.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5c1d71c0 con 0x7f3a5c13e930 2023-11-02T08:36:26.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a537fe640 4 set_mon_vals no callback set 2023-11-02T08:36:26.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a634a7640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:26.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a634a7640 10 monclient: _renew_subs 2023-11-02T08:36:26.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a634a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5c142a40 con 0x7f3a5c13e930 2023-11-02T08:36:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.927+0000 7f3a634a7640 1 librados: init done 2023-11-02T08:36:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.928+0000 7f3a634a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.928+0000 7f3a634a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.928+0000 7f3a634a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:26.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.928+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5c0abd00 con 0x7f3a5c13e930 2023-11-02T08:36:26.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.953+0000 7f3a51ffb640 1 -- 172.21.15.151:0/2050861053 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 4) v1 ==== 88551+0+0 (secure 0 0 0) 0x7f3a54014a30 con 0x7f3a5c13e930 2023-11-02T08:36:26.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.953+0000 7f3a51ffb640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3a4006bd30 0x7f3a4006e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:26.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.953+0000 7f3a61243640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3a4006bd30 0x7f3a4006e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:26.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.953+0000 7f3a61243640 10 monclient: get_auth_request con 0x7f3a4006bd30 auth_method 0 2023-11-02T08:36:26.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.954+0000 7f3a61243640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3a4006bd30 0x7f3a4006e1f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3a58036000 tx=0x7f3a58008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:26.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.954+0000 7f3a51ffb640 10 client.4245.objecter ms_handle_connect 0x7f3a4006bd30 2023-11-02T08:36:26.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.956+0000 7f3a51ffb640 1 -- 172.21.15.151:0/2050861053 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3a5400ce80 con 0x7f3a5c13e930 2023-11-02T08:36:26.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.956+0000 7f3a51ffb640 10 client.4245.objecter ms_dispatch 0x7f3a5c000b90 osd_map(18..18 src has 1..18) v4 2023-11-02T08:36:26.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.956+0000 7f3a51ffb640 3 client.4245.objecter handle_osd_map got epochs [18,18] > 0 2023-11-02T08:36:26.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.956+0000 7f3a51ffb640 3 client.4245.objecter handle_osd_map decoding full epoch 18 2023-11-02T08:36:26.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:26.956+0000 7f3a51ffb640 20 client.4245.objecter dump_active .. 0 homeless 2023-11-02T08:36:27.240 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.237+0000 7f3a51ffb640 1 -- 172.21.15.151:0/2050861053 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f3a540551f0 con 0x7f3a5c13e930 2023-11-02T08:36:27.240 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.237+0000 7f3a51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:27.240 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.237+0000 7f3a51ffb640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:27.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.412+0000 7f3a634a7640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-11-02T08:36:27.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.412+0000 7f3a634a7640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-11-02T08:36:27.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.412+0000 7f3a634a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:27.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.412+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f3a5c0abd00 con 0x7f3a5c13e930 2023-11-02T08:36:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.857+0000 7f3a51ffb640 1 -- 172.21.15.151:0/2050861053 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3a540555c0 con 0x7f3a5c13e930 2023-11-02T08:36:27.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.883+0000 7f3a51ffb640 1 -- 172.21.15.151:0/2050861053 <== mon.0 v2:172.21.15.151:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f3a54058c80 con 0x7f3a5c13e930 2023-11-02T08:36:27.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.883+0000 7f3a51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-11-02T08:36:27.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.883+0000 7f3a51ffb640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-11-02T08:36:27.887 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'rbd' created 2023-11-02T08:36:27.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.885+0000 7f3a634a7640 10 librados: watch_flush enter 2023-11-02T08:36:27.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.885+0000 7f3a634a7640 10 librados: watch_flush exit 2023-11-02T08:36:27.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.885+0000 7f3a634a7640 20 client.4245.objecter shutdown clearing up homeless session... 2023-11-02T08:36:27.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.885+0000 7f3a634a7640 10 client.4245.objecter successfully canceled tick 2023-11-02T08:36:27.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3a4006bd30 msgr2=0x7f3a4006e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:27.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3a4006bd30 0x7f3a4006e1f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3a58036000 tx=0x7f3a58008030 comp rx=0 tx=0).stop 2023-11-02T08:36:27.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 10 monclient: shutdown 2023-11-02T08:36:27.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:27.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c13e930 msgr2=0x7f3a5c0abfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:27.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c13e930 0x7f3a5c0abfe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400ec10 tx=0x7f3a5400c490 comp rx=0 tx=0).stop 2023-11-02T08:36:27.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 shutdown_connections 2023-11-02T08:36:27.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a5c13e930 0x7f3a5c0abfe0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:27.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a5c0b7be0 0x7f3a5c0ab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:27.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3a4006bd30 0x7f3a4006e1f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:27.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 --2- 172.21.15.151:0/2050861053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a5c0a9420 0x7f3a5c1d67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:27.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 >> 172.21.15.151:0/2050861053 conn(0x7f3a5c06c510 msgr2=0x7f3a5c13db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:27.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 shutdown_connections 2023-11-02T08:36:27.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 -- 172.21.15.151:0/2050861053 wait complete. 2023-11-02T08:36:27.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:27.886+0000 7f3a634a7640 1 librados: shutdown 2023-11-02T08:36:27.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-02T08:36:28.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.010+0000 7f7d77fdc640 10 monclient: get_monmap_and_config 2023-11-02T08:36:28.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.010+0000 7f7d77fdc640 10 monclient: build_initial_monmap 2023-11-02T08:36:28.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.010+0000 7f7d77fdc640 10 monclient: monmap: 2023-11-02T08:36:28.015 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:28.015 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:28.015 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:28.011919+0000 2023-11-02T08:36:28.015 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:28.011919+0000 2023-11-02T08:36:28.016 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:28.016 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:28.016 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:28.016 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:28.016 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:28.017 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:28.017 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:28.017 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:28.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.011+0000 7f7d77fdc640 1 Processor -- start 2023-11-02T08:36:28.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.011+0000 7f7d77fdc640 1 -- start start 2023-11-02T08:36:28.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.011+0000 7f7d77fdc640 10 monclient: init 2023-11-02T08:36:28.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:28.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-11-02T08:36:28.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 0x7f7d70141640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:28.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient(hunting): picked mon.noname-b con 0x7f7d700b4100 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:28.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7d700b0cf0 0x7f7d700a8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:28.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient(hunting): picked mon.noname-a con 0x7f7d700b0cf0 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:28.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7d7013e900 0x7f7d7013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:28.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient(hunting): picked mon.noname-c con 0x7f7d7013e900 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:28.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:28.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d7013f2d0 con 0x7f7d700b0cf0 2023-11-02T08:36:28.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:28.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d700b44e0 con 0x7f7d7013e900 2023-11-02T08:36:28.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:28.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d70141b80 con 0x7f7d700b4100 2023-11-02T08:36:28.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:28.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d77fdc640 10 monclient(hunting): authenticate will time out at 1359.368208s 2023-11-02T08:36:28.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.012+0000 7f7d76579640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7d7013e900 0x7f7d7013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:28.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75577640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7d700b0cf0 0x7f7d700a8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:28.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d76579640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7d7013e900 0x7f7d7013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:51022/0 (socket says 172.21.15.151:51022) 2023-11-02T08:36:28.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d76579640 1 -- 172.21.15.151:0/3362107424 learned_addr learned my addr 172.21.15.151:0/3362107424 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:28.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d76579640 10 monclient(hunting): get_auth_request con 0x7f7d7013e900 auth_method 0 2023-11-02T08:36:28.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d76579640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:28.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d76579640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:28.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d76579640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:28.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75577640 10 monclient(hunting): get_auth_request con 0x7f7d700b0cf0 auth_method 0 2023-11-02T08:36:28.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:28.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75577640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:28.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75577640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:28.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75d78640 1 --2- 172.21.15.151:0/3362107424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 0x7f7d70141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:28.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75d78640 10 monclient(hunting): get_auth_request con 0x7f7d700b4100 auth_method 0 2023-11-02T08:36:28.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75d78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:28.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75d78640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:28.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75d78640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:28.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d76579640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:28.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d76579640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:28.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d76579640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:28.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75577640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:28.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:28.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.013+0000 7f7d75577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:28.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.014+0000 7f7d75d78640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:28.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.014+0000 7f7d75d78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:28.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.014+0000 7f7d75d78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:28.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2023-11-02T08:36:28.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 1 -- 172.21.15.151:0/3362107424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 msgr2=0x7f7d70141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:28.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 1 --2- 172.21.15.151:0/3362107424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 0x7f7d70141640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 1 -- 172.21.15.151:0/3362107424 >> v2:172.21.15.151:3300/0 conn(0x7f7d700b0cf0 msgr2=0x7f7d700a8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:28.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 1 --2- 172.21.15.151:0/3362107424 >> v2:172.21.15.151:3300/0 conn(0x7f7d700b0cf0 0x7f7d700a8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 10 monclient: _finish_hunting 0 2023-11-02T08:36:28.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 1 monclient: found mon.noname-c 2023-11-02T08:36:28.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:28.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:28.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.015+0000 7f7d76579640 1 -- 172.21.15.151:0/3362107424 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d700aa360 con 0x7f7d7013e900 2023-11-02T08:36:28.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.017+0000 7f7d76579640 1 --2- 172.21.15.151:0/3362107424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d7013e900 0x7f7d7013ed90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7d700b3030 tx=0x7f7d6802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4bf063cdf6b792a server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:28.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 1 -- 172.21.15.151:0/3362107424 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6802c9e0 con 0x7f7d7013e900 2023-11-02T08:36:28.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:28.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:28.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 10 monclient: dump: 2023-11-02T08:36:28.038 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:28.038 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:28.038 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:28.039 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:28.039 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:28.039 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:28.039 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:28.040 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:28.040 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:28.040 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:28.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 10 monclient: _finish_auth 0 2023-11-02T08:36:28.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 10 monclient: _check_auth_tickets 2023-11-02T08:36:28.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:28.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 1 -- 172.21.15.151:0/3362107424 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6802cb40 con 0x7f7d7013e900 2023-11-02T08:36:28.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:28.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 1 -- 172.21.15.151:0/3362107424 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d680343f0 con 0x7f7d7013e900 2023-11-02T08:36:28.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d77fdc640 5 monclient: authenticate success, global_id 4282 2023-11-02T08:36:28.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d77fdc640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:28.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d77fdc640 4 set_mon_vals no callback set 2023-11-02T08:36:28.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:28.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:28.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d74d76640 10 monclient: dump: 2023-11-02T08:36:28.043 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:28.043 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:28.043 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:28.043 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:28.043 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:28.044 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:28.044 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:28.044 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:28.044 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:28.044 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:28.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d77fdc640 10 monclient: shutdown 2023-11-02T08:36:28.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d77fdc640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:28.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d77fdc640 1 -- 172.21.15.151:0/3362107424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d7013e900 msgr2=0x7f7d7013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:28.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.021+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/3362107424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d7013e900 0x7f7d7013ed90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7d700b3030 tx=0x7f7d6802be40 comp rx=0 tx=0).stop 2023-11-02T08:36:28.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.022+0000 7f7d77fdc640 1 -- 172.21.15.151:0/3362107424 shutdown_connections 2023-11-02T08:36:28.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.022+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/3362107424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d7013e900 0x7f7d7013ed90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.022+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/3362107424 >> v2:172.21.15.151:3300/0 conn(0x7f7d700b0cf0 0x7f7d700a8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.022+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/3362107424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 0x7f7d70141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.022+0000 7f7d77fdc640 1 -- 172.21.15.151:0/3362107424 >> 172.21.15.151:0/3362107424 conn(0x7f7d70065d50 msgr2=0x7f7d70066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:28.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.022+0000 7f7d77fdc640 1 -- 172.21.15.151:0/3362107424 shutdown_connections 2023-11-02T08:36:28.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.022+0000 7f7d77fdc640 1 -- 172.21.15.151:0/3362107424 wait complete. 2023-11-02T08:36:28.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.022+0000 7f7d77fdc640 10 monclient: build_initial_monmap 2023-11-02T08:36:28.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.022+0000 7f7d77fdc640 10 monclient: monmap: 2023-11-02T08:36:28.047 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:28.048 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:28.048 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:28.048 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:28.048 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:28.048 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:28.049 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:28.049 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:28.049 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:28.049 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:28.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 librados: starting msgr at 2023-11-02T08:36:28.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 librados: starting objecter 2023-11-02T08:36:28.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 Processor -- start 2023-11-02T08:36:28.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 -- start start 2023-11-02T08:36:28.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 librados: setting wanted keys 2023-11-02T08:36:28.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 librados: calling monclient init 2023-11-02T08:36:28.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient: init 2023-11-02T08:36:28.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:28.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-02T08:36:28.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d700b0cf0 0x7f7d701c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:28.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient(hunting): picked mon.noname-b con 0x7f7d700b0cf0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:28.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 0x7f7d701d5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:28.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient(hunting): picked mon.noname-c con 0x7f7d700b4100 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:28.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d7013e900 0x7f7d700b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:28.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient(hunting): picked mon.noname-a con 0x7f7d7013e900 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:28.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:28.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d70141550 con 0x7f7d7013e900 2023-11-02T08:36:28.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:28.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d701c96b0 con 0x7f7d700b0cf0 2023-11-02T08:36:28.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:28.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d701d67f0 con 0x7f7d700b4100 2023-11-02T08:36:28.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:28.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.023+0000 7f7d77fdc640 10 monclient(hunting): authenticate will time out at 1359.379578s 2023-11-02T08:36:28.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75577640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 0x7f7d701d5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:28.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75d78640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d700b0cf0 0x7f7d701c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:28.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d76579640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d7013e900 0x7f7d700b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:28.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75d78640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d700b0cf0 0x7f7d701c8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:51032/0 (socket says 172.21.15.151:51032) 2023-11-02T08:36:28.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75d78640 1 -- 172.21.15.151:0/2360121524 learned_addr learned my addr 172.21.15.151:0/2360121524 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:28.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75d78640 10 monclient(hunting): get_auth_request con 0x7f7d700b0cf0 auth_method 0 2023-11-02T08:36:28.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75d78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:28.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75d78640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:28.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75d78640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:28.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75577640 10 monclient(hunting): get_auth_request con 0x7f7d700b4100 auth_method 0 2023-11-02T08:36:28.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:28.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75577640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:28.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d75577640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:28.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d76579640 10 monclient(hunting): get_auth_request con 0x7f7d7013e900 auth_method 0 2023-11-02T08:36:28.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d76579640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:28.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d76579640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:28.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.024+0000 7f7d76579640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:28.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.025+0000 7f7d75d78640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:28.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.025+0000 7f7d75d78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:28.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.025+0000 7f7d75d78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:28.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.025+0000 7f7d75577640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:28.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.025+0000 7f7d75577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:28.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.025+0000 7f7d75577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:28.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.025+0000 7f7d76579640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:28.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.025+0000 7f7d76579640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:28.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.025+0000 7f7d76579640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:28.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 10 monclient(hunting): handle_auth_done global_id 4285 payload 824 2023-11-02T08:36:28.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 1 -- 172.21.15.151:0/2360121524 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 msgr2=0x7f7d701d5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:28.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 0x7f7d701d5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 1 -- 172.21.15.151:0/2360121524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d7013e900 msgr2=0x7f7d700b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:28.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d7013e900 0x7f7d700b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 10 monclient: _finish_hunting 0 2023-11-02T08:36:28.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 1 monclient: found mon.noname-b 2023-11-02T08:36:28.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:28.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:28.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.027+0000 7f7d75d78640 1 -- 172.21.15.151:0/2360121524 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68009b30 con 0x7f7d700b0cf0 2023-11-02T08:36:28.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.030+0000 7f7d75d78640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d700b0cf0 0x7f7d701c8a80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00e9b0 tx=0x7f7d6c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:28.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.030+0000 7f7d667fc640 10 client.?.objecter ms_handle_connect 0x7f7d700b0cf0 2023-11-02T08:36:28.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.030+0000 7f7d667fc640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:28.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 1 -- 172.21.15.151:0/2360121524 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c009670 con 0x7f7d700b0cf0 2023-11-02T08:36:28.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:28.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:28.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 10 monclient: dump: 2023-11-02T08:36:28.065 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:28.065 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:28.065 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:28.066 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:28.066 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:28.066 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:28.066 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:28.066 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:28.067 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:28.067 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:28.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:28.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:28.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:28.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 1 -- 172.21.15.151:0/2360121524 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6c00c9a0 con 0x7f7d700b0cf0 2023-11-02T08:36:28.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:28.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 1 -- 172.21.15.151:0/2360121524 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c00cd50 con 0x7f7d700b0cf0 2023-11-02T08:36:28.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d77fdc640 5 monclient: authenticate success, global_id 4285 2023-11-02T08:36:28.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:28.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:28.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d667fc640 10 monclient: dump: 2023-11-02T08:36:28.069 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:28.069 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:28.069 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:28.069 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:28.070 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:28.070 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:28.070 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:28.071 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:28.071 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:28.071 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:28.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d77fdc640 10 monclient: _renew_subs 2023-11-02T08:36:28.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d67fff640 4 set_mon_vals no callback set 2023-11-02T08:36:28.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d77fdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:28.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d701d64d0 con 0x7f7d700b0cf0 2023-11-02T08:36:28.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d77fdc640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:28.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d77fdc640 10 monclient: _renew_subs 2023-11-02T08:36:28.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d77fdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:28.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d700b52d0 con 0x7f7d700b0cf0 2023-11-02T08:36:28.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.033+0000 7f7d77fdc640 1 librados: init done 2023-11-02T08:36:28.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.034+0000 7f7d77fdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:28.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.034+0000 7f7d77fdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:28.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.034+0000 7f7d77fdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:28.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.034+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7004e040 con 0x7f7d700b0cf0 2023-11-02T08:36:28.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.052+0000 7f7d667fc640 1 -- 172.21.15.151:0/2360121524 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7d6c013470 con 0x7f7d700b0cf0 2023-11-02T08:36:28.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.052+0000 7f7d667fc640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7d5406bdc0 0x7f7d5406e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:28.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.053+0000 7f7d75577640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7d5406bdc0 0x7f7d5406e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:28.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.053+0000 7f7d75577640 10 monclient: get_auth_request con 0x7f7d5406bdc0 auth_method 0 2023-11-02T08:36:28.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.053+0000 7f7d75577640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7d5406bdc0 0x7f7d5406e280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d7013d920 tx=0x7f7d60009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:28.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.053+0000 7f7d667fc640 10 client.4285.objecter ms_handle_connect 0x7f7d5406bdc0 2023-11-02T08:36:28.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.056+0000 7f7d667fc640 1 -- 172.21.15.151:0/2360121524 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f7d6c058600 con 0x7f7d700b0cf0 2023-11-02T08:36:28.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.056+0000 7f7d667fc640 10 client.4285.objecter ms_dispatch 0x7f7d70000b90 osd_map(19..19 src has 1..19) v4 2023-11-02T08:36:28.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.056+0000 7f7d667fc640 3 client.4285.objecter handle_osd_map got epochs [19,19] > 0 2023-11-02T08:36:28.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.056+0000 7f7d667fc640 3 client.4285.objecter handle_osd_map decoding full epoch 19 2023-11-02T08:36:28.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.056+0000 7f7d667fc640 20 client.4285.objecter dump_active .. 0 homeless 2023-11-02T08:36:28.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.306+0000 7f7d667fc640 1 -- 172.21.15.151:0/2360121524 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f7d6c054940 con 0x7f7d700b0cf0 2023-11-02T08:36:28.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.306+0000 7f7d667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:28.310 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.306+0000 7f7d667fc640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:28.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.493+0000 7f7d77fdc640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-11-02T08:36:28.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.493+0000 7f7d77fdc640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-11-02T08:36:28.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.493+0000 7f7d77fdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:28.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.493+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7d701d69b0 con 0x7f7d700b0cf0 2023-11-02T08:36:28.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.860+0000 7f7d667fc640 1 -- 172.21.15.151:0/2360121524 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v20) v1 ==== 173+0+0 (secure 0 0 0) 0x7f7d6c0210c0 con 0x7f7d700b0cf0 2023-11-02T08:36:28.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.860+0000 7f7d667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-11-02T08:36:28.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.860+0000 7f7d667fc640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-11-02T08:36:28.864 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-02T08:36:28.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.862+0000 7f7d77fdc640 10 librados: watch_flush enter 2023-11-02T08:36:28.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.862+0000 7f7d77fdc640 10 librados: watch_flush exit 2023-11-02T08:36:28.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.862+0000 7f7d77fdc640 20 client.4285.objecter shutdown clearing up homeless session... 2023-11-02T08:36:28.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.862+0000 7f7d77fdc640 10 client.4285.objecter successfully canceled tick 2023-11-02T08:36:28.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.862+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7d5406bdc0 msgr2=0x7f7d5406e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:28.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.862+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7d5406bdc0 0x7f7d5406e280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d7013d920 tx=0x7f7d60009190 comp rx=0 tx=0).stop 2023-11-02T08:36:28.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 10 monclient: shutdown 2023-11-02T08:36:28.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:28.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d700b0cf0 msgr2=0x7f7d701c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:28.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d700b0cf0 0x7f7d701c8a80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00e9b0 tx=0x7f7d6c00ee70 comp rx=0 tx=0).stop 2023-11-02T08:36:28.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 shutdown_connections 2023-11-02T08:36:28.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d7013e900 0x7f7d700b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d700b4100 0x7f7d701d5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7d5406bdc0 0x7f7d5406e280 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 --2- 172.21.15.151:0/2360121524 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d700b0cf0 0x7f7d701c8a80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:28.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 >> 172.21.15.151:0/2360121524 conn(0x7f7d70065d50 msgr2=0x7f7d700b76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:28.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 shutdown_connections 2023-11-02T08:36:28.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 -- 172.21.15.151:0/2360121524 wait complete. 2023-11-02T08:36:28.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:28.863+0000 7f7d77fdc640 1 librados: shutdown 2023-11-02T08:36:28.881 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-02T08:36:28.882 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-02T08:36:28.883 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-02T08:36:28.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-02T08:36:28.884 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-02T08:36:29.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.011+0000 7f63570f7640 10 monclient: get_monmap_and_config 2023-11-02T08:36:29.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.011+0000 7f63570f7640 10 monclient: build_initial_monmap 2023-11-02T08:36:29.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.011+0000 7f63570f7640 10 monclient: monmap: 2023-11-02T08:36:29.016 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:29.016 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:29.016 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:29.012742+0000 2023-11-02T08:36:29.017 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:29.012742+0000 2023-11-02T08:36:29.017 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:29.017 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:29.017 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:29.017 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:29.018 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:29.018 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:29.018 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:29.018 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:29.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.012+0000 7f63570f7640 1 Processor -- start 2023-11-02T08:36:29.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.012+0000 7f63570f7640 1 -- start start 2023-11-02T08:36:29.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.012+0000 7f63570f7640 10 monclient: init 2023-11-02T08:36:29.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:29.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-11-02T08:36:29.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63500b4260 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:29.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6350141730 0x7f6350141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:29.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 10 monclient(hunting): picked mon.noname-a con 0x7f6350141730 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:29.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f635013e520 0x7f635013e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:29.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 10 monclient(hunting): picked mon.noname-b con 0x7f635013e520 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:29.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63500a9210 con 0x7f63500b4260 2023-11-02T08:36:29.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:29.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63500aa550 con 0x7f6350141730 2023-11-02T08:36:29.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:29.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63500a8f80 con 0x7f635013e520 2023-11-02T08:36:29.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:29.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.013+0000 7f63570f7640 10 monclient(hunting): authenticate will time out at 1360.369189s 2023-11-02T08:36:29.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f634ffff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6350141730 0x7f6350141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:29.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f6355694640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f635013e520 0x7f635013e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:29.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f634ffff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6350141730 0x7f6350141ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47908/0 (socket says 172.21.15.151:47908) 2023-11-02T08:36:29.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f634ffff640 1 -- 172.21.15.151:0/3046348399 learned_addr learned my addr 172.21.15.151:0/3046348399 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:29.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f634ffff640 10 monclient(hunting): get_auth_request con 0x7f6350141730 auth_method 0 2023-11-02T08:36:29.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f634ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:29.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f634ffff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:29.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f634ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:29.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f6355694640 10 monclient(hunting): get_auth_request con 0x7f635013e520 auth_method 0 2023-11-02T08:36:29.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f6355694640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:29.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f6355694640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:29.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.014+0000 7f6355694640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:29.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.015+0000 7f6355694640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:29.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.015+0000 7f6355694640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:29.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.015+0000 7f6355694640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:29.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.015+0000 7f634ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:29.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.015+0000 7f634ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:29.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.015+0000 7f634ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:29.028 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.016+0000 7f6355694640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2023-11-02T08:36:29.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.016+0000 7f6355694640 1 -- 172.21.15.151:0/3046348399 >> v2:172.21.15.151:3300/0 conn(0x7f6350141730 msgr2=0x7f6350141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:29.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.017+0000 7f6355694640 1 --2- 172.21.15.151:0/3046348399 >> v2:172.21.15.151:3300/0 conn(0x7f6350141730 0x7f6350141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:29.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.017+0000 7f6355694640 1 -- 172.21.15.151:0/3046348399 >> v1:172.21.15.151:6789/0 conn(0x7f63500b4260 legacy=0x7f63500a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:29.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.017+0000 7f6355694640 10 monclient: _finish_hunting 0 2023-11-02T08:36:29.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.017+0000 7f6355694640 1 monclient: found mon.noname-b 2023-11-02T08:36:29.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.017+0000 7f6355694640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:29.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.017+0000 7f6355694640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:29.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.017+0000 7f6355694640 1 -- 172.21.15.151:0/3046348399 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63500aa750 con 0x7f635013e520 2023-11-02T08:36:29.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.019+0000 7f6355694640 1 --2- 172.21.15.151:0/3046348399 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f635013e520 0x7f635013e940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6348009dc0 tx=0x7f634802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9e23481dc45759f server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:29.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 1 -- 172.21.15.151:0/3046348399 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634802cbb0 con 0x7f635013e520 2023-11-02T08:36:29.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:29.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:29.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 10 monclient: dump: 2023-11-02T08:36:29.039 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:29.039 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:29.039 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:29.040 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:29.040 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:29.040 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:29.040 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:29.040 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:29.041 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:29.041 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:29.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 10 monclient: _finish_auth 0 2023-11-02T08:36:29.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 10 monclient: _check_auth_tickets 2023-11-02T08:36:29.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:29.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 1 -- 172.21.15.151:0/3046348399 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634802cd10 con 0x7f635013e520 2023-11-02T08:36:29.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:29.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 1 -- 172.21.15.151:0/3046348399 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348034660 con 0x7f635013e520 2023-11-02T08:36:29.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:29.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:29.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f634f7fe640 10 monclient: dump: 2023-11-02T08:36:29.043 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:29.043 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:29.043 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:29.043 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:29.044 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:29.044 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:29.044 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:29.044 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:29.045 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:29.045 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:29.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f63570f7640 5 monclient: authenticate success, global_id 4229 2023-11-02T08:36:29.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f63570f7640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:29.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.023+0000 7f63570f7640 4 set_mon_vals no callback set 2023-11-02T08:36:29.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.024+0000 7f63570f7640 10 monclient: shutdown 2023-11-02T08:36:29.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.024+0000 7f63570f7640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:29.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.024+0000 7f63570f7640 1 -- 172.21.15.151:0/3046348399 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f635013e520 msgr2=0x7f635013e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:29.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.024+0000 7f63570f7640 1 --2- 172.21.15.151:0/3046348399 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f635013e520 0x7f635013e940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6348009dc0 tx=0x7f634802be00 comp rx=0 tx=0).stop 2023-11-02T08:36:29.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.024+0000 7f63570f7640 1 -- 172.21.15.151:0/3046348399 shutdown_connections 2023-11-02T08:36:29.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.024+0000 7f63570f7640 1 --2- 172.21.15.151:0/3046348399 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f635013e520 0x7f635013e940 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:29.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.024+0000 7f63570f7640 1 --2- 172.21.15.151:0/3046348399 >> v2:172.21.15.151:3300/0 conn(0x7f6350141730 0x7f6350141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:29.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.024+0000 7f63570f7640 1 -- 172.21.15.151:0/3046348399 >> 172.21.15.151:0/3046348399 conn(0x7f6350065d50 msgr2=0x7f6350066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:29.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.025+0000 7f63570f7640 1 -- 172.21.15.151:0/3046348399 shutdown_connections 2023-11-02T08:36:29.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.025+0000 7f63570f7640 1 -- 172.21.15.151:0/3046348399 wait complete. 2023-11-02T08:36:29.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.026+0000 7f63570f7640 10 monclient: build_initial_monmap 2023-11-02T08:36:29.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.026+0000 7f63570f7640 10 monclient: monmap: 2023-11-02T08:36:29.049 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:29.049 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:29.049 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:29.050 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:29.050 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:29.050 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:29.050 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:29.050 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:29.051 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:29.051 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:29.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.026+0000 7f63570f7640 1 librados: starting msgr at 2023-11-02T08:36:29.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.026+0000 7f63570f7640 1 librados: starting objecter 2023-11-02T08:36:29.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.026+0000 7f63570f7640 1 Processor -- start 2023-11-02T08:36:29.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.026+0000 7f63570f7640 1 -- start start 2023-11-02T08:36:29.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.026+0000 7f63570f7640 1 librados: setting wanted keys 2023-11-02T08:36:29.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.026+0000 7f63570f7640 1 librados: calling monclient init 2023-11-02T08:36:29.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.026+0000 7f63570f7640 10 monclient: init 2023-11-02T08:36:29.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:29.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-02T08:36:29.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63500b4260 0x7f63501c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:29.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient(hunting): picked mon.noname-c con 0x7f63500b4260 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:29.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635013e520 0x7f635013f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:29.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient(hunting): picked mon.noname-b con 0x7f635013e520 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:29.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350141730 0x7f63501d6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:29.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient(hunting): picked mon.noname-a con 0x7f6350141730 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:29.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:29.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6350140970 con 0x7f6350141730 2023-11-02T08:36:29.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:29.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63501d6800 con 0x7f635013e520 2023-11-02T08:36:29.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:29.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63501d64d0 con 0x7f63500b4260 2023-11-02T08:36:29.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:29.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.027+0000 7f63570f7640 10 monclient(hunting): authenticate will time out at 1360.383730s 2023-11-02T08:36:29.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.028+0000 7f6355694640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350141730 0x7f63501d6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:29.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6354e93640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63500b4260 0x7f63501c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:29.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6355694640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350141730 0x7f63501d6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:47910/0 (socket says 172.21.15.151:47910) 2023-11-02T08:36:29.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6355694640 1 -- 172.21.15.151:0/1025326323 learned_addr learned my addr 172.21.15.151:0/1025326323 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:29.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6355694640 10 monclient(hunting): get_auth_request con 0x7f6350141730 auth_method 0 2023-11-02T08:36:29.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6355694640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:29.056 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6355694640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:29.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6355694640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:29.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f634ffff640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635013e520 0x7f635013f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:29.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6354e93640 10 monclient(hunting): get_auth_request con 0x7f63500b4260 auth_method 0 2023-11-02T08:36:29.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6354e93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:29.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6354e93640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:29.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6354e93640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:29.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6355694640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:29.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6355694640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:29.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.029+0000 7f6355694640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:29.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.030+0000 7f634ffff640 10 monclient(hunting): get_auth_request con 0x7f635013e520 auth_method 0 2023-11-02T08:36:29.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.030+0000 7f634ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:29.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.030+0000 7f634ffff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:29.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.030+0000 7f634ffff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:29.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.030+0000 7f6354e93640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:29.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.030+0000 7f6354e93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:29.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.030+0000 7f6354e93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:29.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f634ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:29.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f634ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:29.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f634ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:29.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2023-11-02T08:36:29.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 1 -- 172.21.15.151:0/1025326323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63500b4260 msgr2=0x7f63501c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:29.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63500b4260 0x7f63501c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:29.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 1 -- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635013e520 msgr2=0x7f635013f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:29.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635013e520 0x7f635013f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:29.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 10 monclient: _finish_hunting 0 2023-11-02T08:36:29.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 1 monclient: found mon.noname-a 2023-11-02T08:36:29.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:29.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:29.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.034+0000 7f6355694640 1 -- 172.21.15.151:0/1025326323 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6348009a70 con 0x7f6350141730 2023-11-02T08:36:29.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.037+0000 7f6355694640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350141730 0x7f63501d6d20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6348009d90 tx=0x7f6348004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:29.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.037+0000 7f634d7fa640 10 client.?.objecter ms_handle_connect 0x7f6350141730 2023-11-02T08:36:29.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.037+0000 7f634d7fa640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:29.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 1 -- 172.21.15.151:0/1025326323 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348040650 con 0x7f6350141730 2023-11-02T08:36:29.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:29.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 10 monclient: dump: 2023-11-02T08:36:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:29.067 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:29.068 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:29.068 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:29.068 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:29.069 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:29.069 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:29.069 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:29.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 10 monclient: _finish_auth 0 2023-11-02T08:36:29.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 10 monclient: _check_auth_tickets 2023-11-02T08:36:29.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:29.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 1 -- 172.21.15.151:0/1025326323 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63480407b0 con 0x7f6350141730 2023-11-02T08:36:29.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:29.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 1 -- 172.21.15.151:0/1025326323 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348040b60 con 0x7f6350141730 2023-11-02T08:36:29.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:29.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:29.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634d7fa640 10 monclient: dump: 2023-11-02T08:36:29.071 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:29.071 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:29.071 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:29.072 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:29.072 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:29.072 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:29.072 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:29.072 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:29.073 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:29.073 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:29.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.042+0000 7f634effd640 4 set_mon_vals no callback set 2023-11-02T08:36:29.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.043+0000 7f63570f7640 5 monclient: authenticate success, global_id 4260 2023-11-02T08:36:29.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.043+0000 7f63570f7640 10 monclient: _renew_subs 2023-11-02T08:36:29.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.043+0000 7f63570f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:29.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.043+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63501413f0 con 0x7f6350141730 2023-11-02T08:36:29.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.043+0000 7f63570f7640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:29.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.043+0000 7f63570f7640 10 monclient: _renew_subs 2023-11-02T08:36:29.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.043+0000 7f63570f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:29.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.043+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63500b0490 con 0x7f6350141730 2023-11-02T08:36:29.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.043+0000 7f63570f7640 1 librados: init done 2023-11-02T08:36:29.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.044+0000 7f63570f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:29.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.045+0000 7f63570f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:29.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.045+0000 7f63570f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:29.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.045+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6350067920 con 0x7f6350141730 2023-11-02T08:36:29.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.069+0000 7f634d7fa640 1 -- 172.21.15.151:0/1025326323 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6348040cc0 con 0x7f6350141730 2023-11-02T08:36:29.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.070+0000 7f634d7fa640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f633406bdc0 0x7f633406e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:29.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.070+0000 7f6354e93640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f633406bdc0 0x7f633406e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:29.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.070+0000 7f6354e93640 10 monclient: get_auth_request con 0x7f633406bdc0 auth_method 0 2023-11-02T08:36:29.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.071+0000 7f6354e93640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f633406bdc0 0x7f633406e280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6344000f60 tx=0x7f6344009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:29.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.071+0000 7f634d7fa640 10 client.4260.objecter ms_handle_connect 0x7f633406bdc0 2023-11-02T08:36:29.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.380+0000 7f634d7fa640 1 -- 172.21.15.151:0/1025326323 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f634807ad20 con 0x7f6350141730 2023-11-02T08:36:29.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.380+0000 7f634d7fa640 10 client.4260.objecter ms_dispatch 0x7f6350000b90 osd_map(20..20 src has 1..20) v4 2023-11-02T08:36:29.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.380+0000 7f634d7fa640 3 client.4260.objecter handle_osd_map got epochs [20,20] > 0 2023-11-02T08:36:29.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.380+0000 7f634d7fa640 3 client.4260.objecter handle_osd_map decoding full epoch 20 2023-11-02T08:36:29.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.380+0000 7f634d7fa640 20 client.4260.objecter dump_active .. 0 homeless 2023-11-02T08:36:29.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.380+0000 7f634d7fa640 1 -- 172.21.15.151:0/1025326323 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f6348077060 con 0x7f6350141730 2023-11-02T08:36:29.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.380+0000 7f634d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:29.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.380+0000 7f634d7fa640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:29.454 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:01:10.345 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:29.544 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:10.419 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:29.554 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:01:10.445 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:29.592 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:01:10.483 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:29.598 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:10.472 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:29.631 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:01:10.521 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:29.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.658+0000 7f63570f7640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-11-02T08:36:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.658+0000 7f63570f7640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-11-02T08:36:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.658+0000 7f63570f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.658+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f63501d69c0 con 0x7f6350141730 2023-11-02T08:36:29.826 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:10.702 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:29.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.869+0000 7f634d7fa640 1 -- 172.21.15.151:0/1025326323 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+169969 (secure 0 0 0) 0x7f6348044090 con 0x7f6350141730 2023-11-02T08:36:29.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.869+0000 7f634d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-11-02T08:36:29.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.869+0000 7f634d7fa640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:29.875 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:36:29.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.872+0000 7f63570f7640 10 librados: watch_flush enter 2023-11-02T08:36:29.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.872+0000 7f63570f7640 10 librados: watch_flush exit 2023-11-02T08:36:29.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.872+0000 7f63570f7640 20 client.4260.objecter shutdown clearing up homeless session... 2023-11-02T08:36:29.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.872+0000 7f63570f7640 10 client.4260.objecter successfully canceled tick 2023-11-02T08:36:29.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.872+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f633406bdc0 msgr2=0x7f633406e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:29.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.872+0000 7f63570f7640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f633406bdc0 0x7f633406e280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6344000f60 tx=0x7f6344009040 comp rx=0 tx=0).stop 2023-11-02T08:36:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 10 monclient: shutdown 2023-11-02T08:36:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350141730 msgr2=0x7f63501d6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350141730 0x7f63501d6d20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6348009d90 tx=0x7f6348004a40 comp rx=0 tx=0).stop 2023-11-02T08:36:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 shutdown_connections 2023-11-02T08:36:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6350141730 0x7f63501d6d20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f633406bdc0 0x7f633406e280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f635013e520 0x7f635013f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 --2- 172.21.15.151:0/1025326323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63500b4260 0x7f63501c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 >> 172.21.15.151:0/1025326323 conn(0x7f6350065d50 msgr2=0x7f63500b7880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 shutdown_connections 2023-11-02T08:36:29.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 -- 172.21.15.151:0/1025326323 wait complete. 2023-11-02T08:36:29.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:29.873+0000 7f63570f7640 1 librados: shutdown 2023-11-02T08:36:29.892 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":5,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":85498},{"type":"v1","addr":"172.21.15.178:6801","nonce":85498}]},"active_addr":"172.21.15.178:6801/85498","active_change":"2023-11-02T08:35:36.624871+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4100,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":3744335678}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":3642968593}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":879080663}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":3216171022}]}]} 2023-11-02T08:36:29.898 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-02T08:36:29.899 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-02T08:36:29.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-02T08:36:29.923 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:01:10.813 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:30.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.026+0000 7f411e43a640 10 monclient: get_monmap_and_config 2023-11-02T08:36:30.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.026+0000 7f411e43a640 10 monclient: build_initial_monmap 2023-11-02T08:36:30.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.026+0000 7f411e43a640 10 monclient: monmap: 2023-11-02T08:36:30.030 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:30.031 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:30.031 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:30.027971+0000 2023-11-02T08:36:30.031 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:30.027971+0000 2023-11-02T08:36:30.031 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:30.031 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.032 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:30.032 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:30.032 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:30.032 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:30.032 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:30.033 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.027+0000 7f411e43a640 1 Processor -- start 2023-11-02T08:36:30.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.027+0000 7f411e43a640 1 -- start start 2023-11-02T08:36:30.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.027+0000 7f411e43a640 10 monclient: init 2023-11-02T08:36:30.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:30.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-11-02T08:36:30.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41180b4260 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:30.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4118141730 0x7f4118141b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 10 monclient(hunting): picked mon.noname-b con 0x7f4118141730 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:30.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f411813e520 0x7f411813e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 10 monclient(hunting): picked mon.noname-c con 0x7f411813e520 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:30.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41180a9210 con 0x7f41180b4260 2023-11-02T08:36:30.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41180aa550 con 0x7f411813e520 2023-11-02T08:36:30.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41180a8f80 con 0x7f4118141730 2023-11-02T08:36:30.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:30.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411e43a640 10 monclient(hunting): authenticate will time out at 1361.384298s 2023-11-02T08:36:30.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.028+0000 7f411c9d7640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f411813e520 0x7f411813e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:30.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.029+0000 7f4117fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f41180b4260 0x7f41180a8870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:54028/0 (socket says 172.21.15.151:54028) 2023-11-02T08:36:30.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.029+0000 7f4117fff640 1 -- 172.21.15.151:0/2740523540 learned_addr learned my addr 172.21.15.151:0/2740523540 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:30.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.029+0000 7f411c9d7640 10 monclient(hunting): get_auth_request con 0x7f411813e520 auth_method 0 2023-11-02T08:36:30.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.029+0000 7f411c9d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:30.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.029+0000 7f411c9d7640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:30.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.029+0000 7f411c9d7640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:30.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.030+0000 7f411c9d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:30.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.030+0000 7f411c9d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:30.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.030+0000 7f411c9d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:30.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.031+0000 7f411c9d7640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2023-11-02T08:36:30.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.031+0000 7f411c9d7640 1 -- 172.21.15.151:0/2740523540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4118141730 msgr2=0x7f4118141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:30.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.031+0000 7f411c9d7640 1 --2- 172.21.15.151:0/2740523540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4118141730 0x7f4118141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.031+0000 7f411c9d7640 1 -- 172.21.15.151:0/2740523540 >> v1:172.21.15.178:6789/0 conn(0x7f41180b4260 legacy=0x7f41180a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.031+0000 7f411c9d7640 10 monclient: _finish_hunting 0 2023-11-02T08:36:30.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.031+0000 7f411c9d7640 1 monclient: found mon.noname-c 2023-11-02T08:36:30.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.031+0000 7f411c9d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:30.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.031+0000 7f411c9d7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.031+0000 7f411c9d7640 1 -- 172.21.15.151:0/2740523540 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41180aa750 con 0x7f411813e520 2023-11-02T08:36:30.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.034+0000 7f411c9d7640 1 --2- 172.21.15.151:0/2740523540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411813e520 0x7f411813e970 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4110009d90 tx=0x7f411002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=993dda89f2b8329f server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:30.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 1 -- 172.21.15.151:0/2740523540 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411002cbc0 con 0x7f411813e520 2023-11-02T08:36:30.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:30.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:30.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 10 monclient: dump: 2023-11-02T08:36:30.044 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:30.044 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:30.045 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.045 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.045 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:30.045 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.045 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:30.046 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:30.046 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:30.046 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 10 monclient: _finish_auth 0 2023-11-02T08:36:30.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:30.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:30.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 1 -- 172.21.15.151:0/2740523540 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411002cd20 con 0x7f411813e520 2023-11-02T08:36:30.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f411e43a640 5 monclient: authenticate success, global_id 4291 2023-11-02T08:36:30.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f411e43a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:30.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:30.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f411e43a640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:30.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 1 -- 172.21.15.151:0/2740523540 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4110034660 con 0x7f411813e520 2023-11-02T08:36:30.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f411e43a640 4 set_mon_vals no callback set 2023-11-02T08:36:30.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:30.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:30.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.038+0000 7f4116ffd640 10 monclient: dump: 2023-11-02T08:36:30.049 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:30.049 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:30.049 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.049 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.049 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:30.050 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.050 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:30.050 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:30.050 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:30.050 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 10 monclient: shutdown 2023-11-02T08:36:30.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:30.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 1 -- 172.21.15.151:0/2740523540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411813e520 msgr2=0x7f411813e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 1 --2- 172.21.15.151:0/2740523540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411813e520 0x7f411813e970 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4110009d90 tx=0x7f411002be10 comp rx=0 tx=0).stop 2023-11-02T08:36:30.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 1 -- 172.21.15.151:0/2740523540 shutdown_connections 2023-11-02T08:36:30.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 1 --2- 172.21.15.151:0/2740523540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411813e520 0x7f411813e970 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 1 --2- 172.21.15.151:0/2740523540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4118141730 0x7f4118141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 1 -- 172.21.15.151:0/2740523540 >> 172.21.15.151:0/2740523540 conn(0x7f4118065d50 msgr2=0x7f4118066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:30.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 1 -- 172.21.15.151:0/2740523540 shutdown_connections 2023-11-02T08:36:30.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.039+0000 7f411e43a640 1 -- 172.21.15.151:0/2740523540 wait complete. 2023-11-02T08:36:30.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.040+0000 7f411e43a640 10 monclient: build_initial_monmap 2023-11-02T08:36:30.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.040+0000 7f411e43a640 10 monclient: monmap: 2023-11-02T08:36:30.053 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:30.053 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:30.053 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:30.053 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:30.054 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:30.054 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.054 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:30.054 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:30.054 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:30.055 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.041+0000 7f411e43a640 1 librados: starting msgr at 2023-11-02T08:36:30.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.041+0000 7f411e43a640 1 librados: starting objecter 2023-11-02T08:36:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.041+0000 7f411e43a640 1 Processor -- start 2023-11-02T08:36:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.041+0000 7f411e43a640 1 -- start start 2023-11-02T08:36:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.041+0000 7f411e43a640 1 librados: setting wanted keys 2023-11-02T08:36:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.041+0000 7f411e43a640 1 librados: calling monclient init 2023-11-02T08:36:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.041+0000 7f411e43a640 10 monclient: init 2023-11-02T08:36:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:36:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41180b4260 0x7f41181d98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient(hunting): picked mon.noname-a con 0x7f41180b4260 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f411813e520 0x7f41181da060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient(hunting): picked mon.noname-c con 0x7f411813e520 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:30.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4118141730 0x7f411813f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient(hunting): picked mon.noname-b con 0x7f4118141730 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:30.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41181da5a0 con 0x7f41180b4260 2023-11-02T08:36:30.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411813e100 con 0x7f4118141730 2023-11-02T08:36:30.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411813c430 con 0x7f411813e520 2023-11-02T08:36:30.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:30.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.042+0000 7f411e43a640 10 monclient(hunting): authenticate will time out at 1361.398508s 2023-11-02T08:36:30.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.043+0000 7f411c9d7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4118141730 0x7f411813f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:30.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.043+0000 7f411c9d7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4118141730 0x7f411813f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:51064/0 (socket says 172.21.15.151:51064) 2023-11-02T08:36:30.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.043+0000 7f411c9d7640 1 -- 172.21.15.151:0/1865561595 learned_addr learned my addr 172.21.15.151:0/1865561595 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:30.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.043+0000 7f411c9d7640 10 monclient(hunting): get_auth_request con 0x7f4118141730 auth_method 0 2023-11-02T08:36:30.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.043+0000 7f411c9d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:30.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.043+0000 7f411c9d7640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:30.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.043+0000 7f411c9d7640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:30.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.044+0000 7f411c9d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:30.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.044+0000 7f411c9d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:30.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.044+0000 7f411c9d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:30.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 10 monclient(hunting): handle_auth_done global_id 4294 payload 824 2023-11-02T08:36:30.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 1 -- 172.21.15.151:0/1865561595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f411813e520 msgr2=0x7f41181da060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f411813e520 0x7f41181da060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 1 -- 172.21.15.151:0/1865561595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41180b4260 msgr2=0x7f41181d98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41180b4260 0x7f41181d98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 10 monclient: _finish_hunting 0 2023-11-02T08:36:30.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 1 monclient: found mon.noname-b 2023-11-02T08:36:30.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:30.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.046+0000 7f411c9d7640 1 -- 172.21.15.151:0/1865561595 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110009a70 con 0x7f4118141730 2023-11-02T08:36:30.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.048+0000 7f411c9d7640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4118141730 0x7f411813f530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4110009d90 tx=0x7f4110004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:30.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.048+0000 7f4114ff9640 10 client.?.objecter ms_handle_connect 0x7f4118141730 2023-11-02T08:36:30.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.048+0000 7f4114ff9640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:30.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 1 -- 172.21.15.151:0/1865561595 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4110040450 con 0x7f4118141730 2023-11-02T08:36:30.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:30.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:30.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 10 monclient: dump: 2023-11-02T08:36:30.068 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:30.068 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:30.068 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.068 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.069 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:30.069 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.069 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:30.069 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:30.069 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:30.069 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 10 monclient: _finish_auth 0 2023-11-02T08:36:30.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 10 monclient: _check_auth_tickets 2023-11-02T08:36:30.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:30.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 1 -- 172.21.15.151:0/1865561595 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411002ce90 con 0x7f4118141730 2023-11-02T08:36:30.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:30.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 1 -- 172.21.15.151:0/1865561595 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4110040810 con 0x7f4118141730 2023-11-02T08:36:30.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:30.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:30.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f4114ff9640 10 monclient: dump: 2023-11-02T08:36:30.071 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:30.072 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:30.072 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.072 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.072 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:30.072 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.073 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:30.073 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:30.073 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:30.073 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f411e43a640 5 monclient: authenticate success, global_id 4294 2023-11-02T08:36:30.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f411e43a640 10 monclient: _renew_subs 2023-11-02T08:36:30.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f411e43a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411813fa70 con 0x7f4118141730 2023-11-02T08:36:30.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f41167fc640 4 set_mon_vals no callback set 2023-11-02T08:36:30.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f411e43a640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:30.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f411e43a640 10 monclient: _renew_subs 2023-11-02T08:36:30.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f411e43a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4118139160 con 0x7f4118141730 2023-11-02T08:36:30.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.052+0000 7f411e43a640 1 librados: init done 2023-11-02T08:36:30.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.053+0000 7f411e43a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:30.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.053+0000 7f411e43a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:30.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.053+0000 7f411e43a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.053+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4118067920 con 0x7f4118141730 2023-11-02T08:36:30.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.071+0000 7f4114ff9640 1 -- 172.21.15.151:0/1865561595 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4110040970 con 0x7f4118141730 2023-11-02T08:36:30.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.071+0000 7f4114ff9640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f410406bdc0 0x7f410406e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.072+0000 7f4117fff640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f410406bdc0 0x7f410406e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:30.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.072+0000 7f4117fff640 10 monclient: get_auth_request con 0x7f410406bdc0 auth_method 0 2023-11-02T08:36:30.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.072+0000 7f4114ff9640 10 client.4294.objecter ms_handle_connect 0x7f410406bdc0 2023-11-02T08:36:30.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.072+0000 7f4117fff640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f410406bdc0 0x7f410406e280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41080015f0 tx=0x7f4108007a40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:30.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.075+0000 7f4114ff9640 1 -- 172.21.15.151:0/1865561595 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4110047e40 con 0x7f4118141730 2023-11-02T08:36:30.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.075+0000 7f4114ff9640 10 client.4294.objecter ms_dispatch 0x7f4118000b90 osd_map(20..20 src has 1..20) v4 2023-11-02T08:36:30.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.075+0000 7f4114ff9640 3 client.4294.objecter handle_osd_map got epochs [20,20] > 0 2023-11-02T08:36:30.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.075+0000 7f4114ff9640 3 client.4294.objecter handle_osd_map decoding full epoch 20 2023-11-02T08:36:30.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.075+0000 7f4114ff9640 20 client.4294.objecter dump_active .. 0 homeless 2023-11-02T08:36:30.105 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:01:10.995 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:30.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.314+0000 7f4114ff9640 1 -- 172.21.15.151:0/1865561595 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f4110077230 con 0x7f4118141730 2023-11-02T08:36:30.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.314+0000 7f4114ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:30.318 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.314+0000 7f4114ff9640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:30.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.548+0000 7f411e43a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:30.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.548+0000 7f411e43a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:30.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.548+0000 7f411e43a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.548+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4118068b90 con 0x7f4118141730 2023-11-02T08:36:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.593+0000 7f4114ff9640 1 -- 172.21.15.151:0/1865561595 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f411007a850 con 0x7f4118141730 2023-11-02T08:36:30.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.593+0000 7f4114ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:30.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.593+0000 7f4114ff9640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:30.599 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:36:30.599 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":21,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:36:29.873253+0000","last_up_change":"2023-11-02T08:36:19.358160+0000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-02T08:36:21.174515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-02T08:36:27.423766+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":86098},{"type":"v1","addr":"172.21.15.151:6809","nonce":86098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":86098},{"type":"v1","addr":"172.21.15.151:6811","nonce":86098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":86098},{"type":"v1","addr":"172.21.15.151:6815","nonce":86098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":86098},{"type":"v1","addr":"172.21.15.151:6813","nonce":86098}]},"public_addr":"172.21.15.151:6809/86098","cluster_addr":"172.21.15.151:6811/86098","heartbeat_back_addr":"172.21.15.151:6815/86098","heartbeat_front_addr":"172.21.15.151:6813/86098","state":["exists","up"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":86101},{"type":"v1","addr":"172.21.15.151:6801","nonce":86101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":86101},{"type":"v1","addr":"172.21.15.151:6803","nonce":86101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":86101},{"type":"v1","addr":"172.21.15.151:6807","nonce":86101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":86101},{"type":"v1","addr":"172.21.15.151:6805","nonce":86101}]},"public_addr":"172.21.15.151:6801/86101","cluster_addr":"172.21.15.151:6803/86101","heartbeat_back_addr":"172.21.15.151:6807/86101","heartbeat_front_addr":"172.21.15.151:6805/86101","state":["exists","up"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":86100},{"type":"v1","addr":"172.21.15.151:6817","nonce":86100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":86100},{"type":"v1","addr":"172.21.15.151:6819","nonce":86100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":86100},{"type":"v1","addr":"172.21.15.151:6823","nonce":86100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":86100},{"type":"v1","addr":"172.21.15.151:6821","nonce":86100}]},"public_addr":"172.21.15.151:6817/86100","cluster_addr":"172.21.15.151:6819/86100","heartbeat_back_addr":"172.21.15.151:6823/86100","heartbeat_front_addr":"172.21.15.151:6821/86100","state":["exists","up"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":86099},{"type":"v1","addr":"172.21.15.151:6825","nonce":86099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":86099},{"type":"v1","addr":"172.21.15.151:6827","nonce":86099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":86099},{"type":"v1","addr":"172.21.15.151:6831","nonce":86099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":86099},{"type":"v1","addr":"172.21.15.151:6829","nonce":86099}]},"public_addr":"172.21.15.151:6825/86099","cluster_addr":"172.21.15.151:6827/86099","heartbeat_back_addr":"172.21.15.151:6831/86099","heartbeat_front_addr":"172.21.15.151:6829/86099","state":["exists","up"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":86120},{"type":"v1","addr":"172.21.15.178:6811","nonce":86120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":86120},{"type":"v1","addr":"172.21.15.178:6813","nonce":86120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":86120},{"type":"v1","addr":"172.21.15.178:6817","nonce":86120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":86120},{"type":"v1","addr":"172.21.15.178:6815","nonce":86120}]},"public_addr":"172.21.15.178:6811/86120","cluster_addr":"172.21.15.178:6813/86120","heartbeat_back_addr":"172.21.15.178:6817/86120","heartbeat_front_addr":"172.21.15.178:6815/86120","state":["exists","up"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":86122},{"type":"v1","addr":"172.21.15.178:6819","nonce":86122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":86122},{"type":"v1","addr":"172.21.15.178:6821","nonce":86122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":86122},{"type":"v1","addr":"172.21.15.178:6825","nonce":86122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":86122},{"type":"v1","addr":"172.21.15.178:6823","nonce":86122}]},"public_addr":"172.21.15.178:6819/86122","cluster_addr":"172.21.15.178:6821/86122","heartbeat_back_addr":"172.21.15.178:6825/86122","heartbeat_front_addr":"172.21.15.178:6823/86122","state":["exists","up"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":86121},{"type":"v1","addr":"172.21.15.178:6827","nonce":86121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":86121},{"type":"v1","addr":"172.21.15.178:6829","nonce":86121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":86121},{"type":"v1","addr":"172.21.15.178:6833","nonce":86121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":86121},{"type":"v1","addr":"172.21.15.178:6831","nonce":86121}]},"public_addr":"172.21.15.178:6827/86121","cluster_addr":"172.21.15.178:6829/86121","heartbeat_back_addr":"172.21.15.178:6833/86121","heartbeat_front_addr":"172.21.15.178:6831/86121","state":["exists","up"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":86123},{"type":"v1","addr":"172.21.15.178:6803","nonce":86123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":86123},{"type":"v1","addr":"172.21.15.178:6805","nonce":86123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":86123},{"type":"v1","addr":"172.21.15.178:6809","nonce":86123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":86123},{"type":"v1","addr":"172.21.15.178:6807","nonce":86123}]},"public_addr":"172.21.15.178:6803/86123","cluster_addr":"172.21.15.178:6805/86123","heartbeat_back_addr":"172.21.15.178:6809/86123","heartbeat_front_addr":"172.21.15.178:6807/86123","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.010757+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:17.073234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.605027+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.579731+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.490219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:15.935790+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:15.992196+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.123838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:36:30.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 10 librados: watch_flush enter 2023-11-02T08:36:30.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 10 librados: watch_flush exit 2023-11-02T08:36:30.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 20 client.4294.objecter shutdown clearing up homeless session... 2023-11-02T08:36:30.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 10 client.4294.objecter successfully canceled tick 2023-11-02T08:36:30.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f410406bdc0 msgr2=0x7f410406e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f410406bdc0 0x7f410406e280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41080015f0 tx=0x7f4108007a40 comp rx=0 tx=0).stop 2023-11-02T08:36:30.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 10 monclient: shutdown 2023-11-02T08:36:30.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:30.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4118141730 msgr2=0x7f411813f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4118141730 0x7f411813f530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4110009d90 tx=0x7f4110004a40 comp rx=0 tx=0).stop 2023-11-02T08:36:30.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 shutdown_connections 2023-11-02T08:36:30.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f410406bdc0 0x7f410406e280 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4118141730 0x7f411813f530 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f411813e520 0x7f41181da060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 --2- 172.21.15.151:0/1865561595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41180b4260 0x7f41181d98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 >> 172.21.15.151:0/1865561595 conn(0x7f4118065d50 msgr2=0x7f41181423b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:30.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 shutdown_connections 2023-11-02T08:36:30.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.595+0000 7f411e43a640 1 -- 172.21.15.151:0/1865561595 wait complete. 2023-11-02T08:36:30.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.596+0000 7f411e43a640 1 librados: shutdown 2023-11-02T08:36:30.611 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-02T08:36:30.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-02T08:36:30.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.749+0000 7f8edd4b6640 10 monclient: get_monmap_and_config 2023-11-02T08:36:30.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.749+0000 7f8edd4b6640 10 monclient: build_initial_monmap 2023-11-02T08:36:30.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.749+0000 7f8edd4b6640 10 monclient: monmap: 2023-11-02T08:36:30.753 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:30.754 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:30.754 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:30.750604+0000 2023-11-02T08:36:30.754 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:30.750604+0000 2023-11-02T08:36:30.754 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:30.755 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.755 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:30.755 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:30.756 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:30.756 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:30.756 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:30.757 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 1 Processor -- start 2023-11-02T08:36:30.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 1 -- start start 2023-11-02T08:36:30.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient: init 2023-11-02T08:36:30.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:30.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-11-02T08:36:30.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ed80b4170 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:30.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8ed80a8df0 0x7f8ed8141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed80a8df0 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:30.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8ed80b13e0 0x7f8ed813e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient(hunting): picked mon.noname-c con 0x7f8ed80b13e0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:30.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed80aa3d0 con 0x7f8ed80b4170 2023-11-02T08:36:30.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed80b4550 con 0x7f8ed80a8df0 2023-11-02T08:36:30.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed80a92d0 con 0x7f8ed80b13e0 2023-11-02T08:36:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.750+0000 7f8edd4b6640 10 monclient(hunting): authenticate will time out at 1362.106647s 2023-11-02T08:36:30.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.751+0000 7f8ece7fc640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8ed80a8df0 0x7f8ed8141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:30.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.751+0000 7f8ed77fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8ed80b13e0 0x7f8ed813e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:30.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.751+0000 7f8ed77fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8ed80b13e0 0x7f8ed813e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:51078/0 (socket says 172.21.15.151:51078) 2023-11-02T08:36:30.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.751+0000 7f8ed77fe640 1 -- 172.21.15.151:0/96318945 learned_addr learned my addr 172.21.15.151:0/96318945 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:30.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.751+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ed80b13e0 auth_method 0 2023-11-02T08:36:30.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.751+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:30.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.751+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.751+0000 7f8ed77fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ece7fc640 10 monclient(hunting): get_auth_request con 0x7f8ed80a8df0 auth_method 0 2023-11-02T08:36:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ece7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:30.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ece7fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ece7fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:30.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:30.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ece7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:30.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ece7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:30.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.752+0000 7f8ece7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:30.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.753+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2023-11-02T08:36:30.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.753+0000 7f8ed77fe640 1 -- 172.21.15.151:0/96318945 >> v2:172.21.15.151:3300/0 conn(0x7f8ed80a8df0 msgr2=0x7f8ed8141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:30.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.753+0000 7f8ed77fe640 1 --2- 172.21.15.151:0/96318945 >> v2:172.21.15.151:3300/0 conn(0x7f8ed80a8df0 0x7f8ed8141f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.754+0000 7f8ed77fe640 1 -- 172.21.15.151:0/96318945 >> v1:172.21.15.151:6789/0 conn(0x7f8ed80b4170 legacy=0x7f8ed8141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.754+0000 7f8ed77fe640 10 monclient: _finish_hunting 0 2023-11-02T08:36:30.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.754+0000 7f8ed77fe640 1 monclient: found mon.noname-c 2023-11-02T08:36:30.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.754+0000 7f8ed77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:30.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.754+0000 7f8ed77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.754+0000 7f8ed77fe640 1 -- 172.21.15.151:0/96318945 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed8142450 con 0x7f8ed80b13e0 2023-11-02T08:36:30.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.756+0000 7f8ed77fe640 1 --2- 172.21.15.151:0/96318945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80b13e0 0x7f8ed813e340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009e50 tx=0x7f8ed002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb981ab50fa4a1b server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:30.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 1 -- 172.21.15.151:0/96318945 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed002ca80 con 0x7f8ed80b13e0 2023-11-02T08:36:30.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:30.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:30.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 10 monclient: dump: 2023-11-02T08:36:30.787 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:30.787 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:30.787 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.788 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.788 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:30.789 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.789 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:30.789 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:30.790 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:30.791 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:30.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:30.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:30.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 1 -- 172.21.15.151:0/96318945 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed002cbe0 con 0x7f8ed80b13e0 2023-11-02T08:36:30.793 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8edd4b6640 5 monclient: authenticate success, global_id 4297 2023-11-02T08:36:30.793 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8edd4b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:30.793 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:30.793 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8edd4b6640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:30.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 1 -- 172.21.15.151:0/96318945 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed00343f0 con 0x7f8ed80b13e0 2023-11-02T08:36:30.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:30.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8edd4b6640 4 set_mon_vals no callback set 2023-11-02T08:36:30.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:30.794 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.762+0000 7f8ed67fc640 10 monclient: dump: 2023-11-02T08:36:30.795 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:30.795 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:30.796 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.797 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.797 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:30.797 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.798 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:30.798 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:30.798 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:30.798 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 10 monclient: shutdown 2023-11-02T08:36:30.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:30.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 1 -- 172.21.15.151:0/96318945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80b13e0 msgr2=0x7f8ed813e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 1 --2- 172.21.15.151:0/96318945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80b13e0 0x7f8ed813e340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009e50 tx=0x7f8ed002bee0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 1 -- 172.21.15.151:0/96318945 shutdown_connections 2023-11-02T08:36:30.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 1 --2- 172.21.15.151:0/96318945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80b13e0 0x7f8ed813e340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 1 --2- 172.21.15.151:0/96318945 >> v2:172.21.15.151:3300/0 conn(0x7f8ed80a8df0 0x7f8ed8141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 1 -- 172.21.15.151:0/96318945 >> 172.21.15.151:0/96318945 conn(0x7f8ed8065d80 msgr2=0x7f8ed8066190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:30.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 1 -- 172.21.15.151:0/96318945 shutdown_connections 2023-11-02T08:36:30.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.763+0000 7f8edd4b6640 1 -- 172.21.15.151:0/96318945 wait complete. 2023-11-02T08:36:30.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.764+0000 7f8edd4b6640 10 monclient: build_initial_monmap 2023-11-02T08:36:30.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.764+0000 7f8edd4b6640 10 monclient: monmap: 2023-11-02T08:36:30.801 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:30.802 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:30.802 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:30.803 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:30.803 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:30.803 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.803 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:30.803 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:30.804 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:30.804 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.764+0000 7f8edd4b6640 1 librados: starting msgr at 2023-11-02T08:36:30.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.764+0000 7f8edd4b6640 1 librados: starting objecter 2023-11-02T08:36:30.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.764+0000 7f8edd4b6640 1 Processor -- start 2023-11-02T08:36:30.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.764+0000 7f8edd4b6640 1 -- start start 2023-11-02T08:36:30.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.764+0000 7f8edd4b6640 1 librados: setting wanted keys 2023-11-02T08:36:30.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.764+0000 7f8edd4b6640 1 librados: calling monclient init 2023-11-02T08:36:30.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.764+0000 7f8edd4b6640 10 monclient: init 2023-11-02T08:36:30.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:30.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:36:30.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80a8df0 0x7f8ed80ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed80a8df0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:30.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ed80b13e0 0x7f8ed80abd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed80b13e0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:30.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ed80b4170 0x7f8ed81d9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:30.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient(hunting): picked mon.noname-c con 0x7f8ed80b4170 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:30.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.808 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed80ac2a0 con 0x7f8ed80b13e0 2023-11-02T08:36:30.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed80ad3f0 con 0x7f8ed80a8df0 2023-11-02T08:36:30.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:30.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed81455c0 con 0x7f8ed80b4170 2023-11-02T08:36:30.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:30.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8edd4b6640 10 monclient(hunting): authenticate will time out at 1362.121288s 2023-11-02T08:36:30.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8ece7fc640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ed80b13e0 0x7f8ed80abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:30.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.765+0000 7f8ed6ffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80a8df0 0x7f8ed80ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:30.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed77fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ed80b4170 0x7f8ed81d9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:30.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed6ffd640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80a8df0 0x7f8ed80ab2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:51088/0 (socket says 172.21.15.151:51088) 2023-11-02T08:36:30.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed6ffd640 1 -- 172.21.15.151:0/2838159606 learned_addr learned my addr 172.21.15.151:0/2838159606 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:30.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ed80a8df0 auth_method 0 2023-11-02T08:36:30.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:30.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:30.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:30.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ece7fc640 10 monclient(hunting): get_auth_request con 0x7f8ed80b13e0 auth_method 0 2023-11-02T08:36:30.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ece7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:30.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ece7fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:30.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ece7fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:30.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ed80b4170 auth_method 0 2023-11-02T08:36:30.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:30.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:30.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed77fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:30.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:30.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:30.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:30.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ece7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:30.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.766+0000 7f8ece7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:30.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.767+0000 7f8ece7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:30.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.767+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:30.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.767+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:30.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.767+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:30.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2023-11-02T08:36:30.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 1 -- 172.21.15.151:0/2838159606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ed80b4170 msgr2=0x7f8ed81d9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ed80b4170 0x7f8ed81d9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 1 -- 172.21.15.151:0/2838159606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ed80b13e0 msgr2=0x7f8ed80abd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:30.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ed80b13e0 0x7f8ed80abd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:30.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 10 monclient: _finish_hunting 0 2023-11-02T08:36:30.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 1 monclient: found mon.noname-b 2023-11-02T08:36:30.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:30.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.769+0000 7f8ed6ffd640 1 -- 172.21.15.151:0/2838159606 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0009b30 con 0x7f8ed80a8df0 2023-11-02T08:36:30.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.772+0000 7f8ed6ffd640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80a8df0 0x7f8ed80ab2e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800c3c0 tx=0x7f8ec800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:30.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.772+0000 7f8ecffff640 10 client.?.objecter ms_handle_connect 0x7f8ed80a8df0 2023-11-02T08:36:30.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.772+0000 7f8ecffff640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:30.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8ecffff640 1 -- 172.21.15.151:0/2838159606 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec80052d0 con 0x7f8ed80a8df0 2023-11-02T08:36:30.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8ecffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:30.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8ecffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:30.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8ecffff640 10 monclient: dump: 2023-11-02T08:36:30.819 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:30.820 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:30.820 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.820 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:30.820 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:30.821 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:30.821 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:30.821 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:30.821 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:30.821 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:30.821 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8ecffff640 10 monclient: _finish_auth 0 2023-11-02T08:36:30.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8ecffff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:30.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8ecffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:30.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8edd4b6640 5 monclient: authenticate success, global_id 4300 2023-11-02T08:36:30.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8edd4b6640 10 monclient: _renew_subs 2023-11-02T08:36:30.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8edd4b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed81da340 con 0x7f8ed80a8df0 2023-11-02T08:36:30.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8edd4b6640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:30.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8edd4b6640 10 monclient: _renew_subs 2023-11-02T08:36:30.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8edd4b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed81da8a0 con 0x7f8ed80a8df0 2023-11-02T08:36:30.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.774+0000 7f8edd4b6640 1 librados: init done 2023-11-02T08:36:30.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.775+0000 7f8edd4b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:30.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.775+0000 7f8edd4b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:30.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.775+0000 7f8edd4b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:30.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.775+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed804e040 con 0x7f8ed80a8df0 2023-11-02T08:36:30.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.988+0000 7f8ecffff640 1 -- 172.21.15.151:0/2838159606 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec8005430 con 0x7f8ed80a8df0 2023-11-02T08:36:30.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.988+0000 7f8ecffff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:30.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:30.988+0000 7f8ed5ffb640 4 set_mon_vals no callback set 2023-11-02T08:36:31.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.045+0000 7f8ecffff640 1 -- 172.21.15.151:0/2838159606 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8007250 con 0x7f8ed80a8df0 2023-11-02T08:36:31.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.045+0000 7f8ecffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.045+0000 7f8ecffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:31.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.045+0000 7f8ecffff640 10 monclient: dump: 2023-11-02T08:36:31.049 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.050 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.050 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.050 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.050 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.050 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.051 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.051 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.051 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.051 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.046+0000 7f8ecffff640 1 -- 172.21.15.151:0/2838159606 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8ec8007470 con 0x7f8ed80a8df0 2023-11-02T08:36:31.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.046+0000 7f8ecffff640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8eb806bdc0 0x7f8eb806e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.046+0000 7f8ece7fc640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8eb806bdc0 0x7f8eb806e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.046+0000 7f8ece7fc640 10 monclient: get_auth_request con 0x7f8eb806bdc0 auth_method 0 2023-11-02T08:36:31.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.047+0000 7f8ecffff640 10 client.4300.objecter ms_handle_connect 0x7f8eb806bdc0 2023-11-02T08:36:31.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.047+0000 7f8ece7fc640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8eb806bdc0 0x7f8eb806e280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ed813b010 tx=0x7f8ec0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.048+0000 7f8ecffff640 1 -- 172.21.15.151:0/2838159606 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8ec8057440 con 0x7f8ed80a8df0 2023-11-02T08:36:31.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.048+0000 7f8ecffff640 10 client.4300.objecter ms_dispatch 0x7f8ed8000b90 osd_map(21..21 src has 1..21) v4 2023-11-02T08:36:31.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.048+0000 7f8ecffff640 3 client.4300.objecter handle_osd_map got epochs [21,21] > 0 2023-11-02T08:36:31.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.048+0000 7f8ecffff640 3 client.4300.objecter handle_osd_map decoding full epoch 21 2023-11-02T08:36:31.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.048+0000 7f8ecffff640 20 client.4300.objecter dump_active .. 0 homeless 2023-11-02T08:36:31.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.070+0000 7f8ecffff640 1 -- 172.21.15.151:0/2838159606 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f8ec8053780 con 0x7f8ed80a8df0 2023-11-02T08:36:31.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.070+0000 7f8ecffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:31.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.070+0000 7f8ecffff640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:31.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.243+0000 7f8edd4b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:31.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.243+0000 7f8edd4b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:31.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.243+0000 7f8edd4b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.247 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.243+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ed80a9b30 con 0x7f8ed80a8df0 2023-11-02T08:36:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.271+0000 7f8ecffff640 1 -- 172.21.15.151:0/2838159606 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f8ec8020090 con 0x7f8ed80a8df0 2023-11-02T08:36:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.271+0000 7f8ecffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-11-02T08:36:31.275 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.271+0000 7f8ecffff640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:36:31.275 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":21,"fsid":"4d1a52ed-5f96-4808-8336-f35436fa00ba","created":"2023-11-02T08:35:31.189936+0000","modified":"2023-11-02T08:36:29.873253+0000","last_up_change":"2023-11-02T08:36:19.358160+0000","last_in_change":"2023-11-02T08:35:37.853543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-02T08:36:21.174515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-02T08:36:27.423766+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fed323c8-4fdc-46c5-bf48-ba99f939a935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":86098},{"type":"v1","addr":"172.21.15.151:6809","nonce":86098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":86098},{"type":"v1","addr":"172.21.15.151:6811","nonce":86098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":86098},{"type":"v1","addr":"172.21.15.151:6815","nonce":86098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":86098},{"type":"v1","addr":"172.21.15.151:6813","nonce":86098}]},"public_addr":"172.21.15.151:6809/86098","cluster_addr":"172.21.15.151:6811/86098","heartbeat_back_addr":"172.21.15.151:6815/86098","heartbeat_front_addr":"172.21.15.151:6813/86098","state":["exists","up"]},{"osd":1,"uuid":"606297f6-206d-441d-99cd-3f50f334b5ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":86101},{"type":"v1","addr":"172.21.15.151:6801","nonce":86101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":86101},{"type":"v1","addr":"172.21.15.151:6803","nonce":86101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":86101},{"type":"v1","addr":"172.21.15.151:6807","nonce":86101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":86101},{"type":"v1","addr":"172.21.15.151:6805","nonce":86101}]},"public_addr":"172.21.15.151:6801/86101","cluster_addr":"172.21.15.151:6803/86101","heartbeat_back_addr":"172.21.15.151:6807/86101","heartbeat_front_addr":"172.21.15.151:6805/86101","state":["exists","up"]},{"osd":2,"uuid":"070974a7-a457-460e-a323-d4685aa2950e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":86100},{"type":"v1","addr":"172.21.15.151:6817","nonce":86100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":86100},{"type":"v1","addr":"172.21.15.151:6819","nonce":86100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":86100},{"type":"v1","addr":"172.21.15.151:6823","nonce":86100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":86100},{"type":"v1","addr":"172.21.15.151:6821","nonce":86100}]},"public_addr":"172.21.15.151:6817/86100","cluster_addr":"172.21.15.151:6819/86100","heartbeat_back_addr":"172.21.15.151:6823/86100","heartbeat_front_addr":"172.21.15.151:6821/86100","state":["exists","up"]},{"osd":3,"uuid":"cf1a441c-66d4-4ef2-9e9b-24afaa41d6d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":86099},{"type":"v1","addr":"172.21.15.151:6825","nonce":86099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":86099},{"type":"v1","addr":"172.21.15.151:6827","nonce":86099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":86099},{"type":"v1","addr":"172.21.15.151:6831","nonce":86099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":86099},{"type":"v1","addr":"172.21.15.151:6829","nonce":86099}]},"public_addr":"172.21.15.151:6825/86099","cluster_addr":"172.21.15.151:6827/86099","heartbeat_back_addr":"172.21.15.151:6831/86099","heartbeat_front_addr":"172.21.15.151:6829/86099","state":["exists","up"]},{"osd":4,"uuid":"cf07e867-481c-4d45-945a-88e8e08d186d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":86120},{"type":"v1","addr":"172.21.15.178:6811","nonce":86120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":86120},{"type":"v1","addr":"172.21.15.178:6813","nonce":86120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":86120},{"type":"v1","addr":"172.21.15.178:6817","nonce":86120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":86120},{"type":"v1","addr":"172.21.15.178:6815","nonce":86120}]},"public_addr":"172.21.15.178:6811/86120","cluster_addr":"172.21.15.178:6813/86120","heartbeat_back_addr":"172.21.15.178:6817/86120","heartbeat_front_addr":"172.21.15.178:6815/86120","state":["exists","up"]},{"osd":5,"uuid":"da85ceeb-7ee7-44cb-8460-4d0aad9c6e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":86122},{"type":"v1","addr":"172.21.15.178:6819","nonce":86122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":86122},{"type":"v1","addr":"172.21.15.178:6821","nonce":86122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":86122},{"type":"v1","addr":"172.21.15.178:6825","nonce":86122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":86122},{"type":"v1","addr":"172.21.15.178:6823","nonce":86122}]},"public_addr":"172.21.15.178:6819/86122","cluster_addr":"172.21.15.178:6821/86122","heartbeat_back_addr":"172.21.15.178:6825/86122","heartbeat_front_addr":"172.21.15.178:6823/86122","state":["exists","up"]},{"osd":6,"uuid":"bfab5b18-727d-4c9f-a9f0-30d624e82282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":86121},{"type":"v1","addr":"172.21.15.178:6827","nonce":86121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":86121},{"type":"v1","addr":"172.21.15.178:6829","nonce":86121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":86121},{"type":"v1","addr":"172.21.15.178:6833","nonce":86121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":86121},{"type":"v1","addr":"172.21.15.178:6831","nonce":86121}]},"public_addr":"172.21.15.178:6827/86121","cluster_addr":"172.21.15.178:6829/86121","heartbeat_back_addr":"172.21.15.178:6833/86121","heartbeat_front_addr":"172.21.15.178:6831/86121","state":["exists","up"]},{"osd":7,"uuid":"dedf0003-4a0c-4672-81f3-3808816fca03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":86123},{"type":"v1","addr":"172.21.15.178:6803","nonce":86123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":86123},{"type":"v1","addr":"172.21.15.178:6805","nonce":86123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":86123},{"type":"v1","addr":"172.21.15.178:6809","nonce":86123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":86123},{"type":"v1","addr":"172.21.15.178:6807","nonce":86123}]},"public_addr":"172.21.15.178:6803/86123","cluster_addr":"172.21.15.178:6805/86123","heartbeat_back_addr":"172.21.15.178:6809/86123","heartbeat_front_addr":"172.21.15.178:6807/86123","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.010757+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:17.073234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.605027+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.579731+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.490219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:15.935790+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:15.992196+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-02T08:36:16.123838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-02T08:36:31.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 10 librados: watch_flush enter 2023-11-02T08:36:31.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 10 librados: watch_flush exit 2023-11-02T08:36:31.277 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 20 client.4300.objecter shutdown clearing up homeless session... 2023-11-02T08:36:31.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 10 client.4300.objecter successfully canceled tick 2023-11-02T08:36:31.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8eb806bdc0 msgr2=0x7f8eb806e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8eb806bdc0 0x7f8eb806e280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ed813b010 tx=0x7f8ec0009190 comp rx=0 tx=0).stop 2023-11-02T08:36:31.278 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 10 monclient: shutdown 2023-11-02T08:36:31.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:31.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80a8df0 msgr2=0x7f8ed80ab2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80a8df0 0x7f8ed80ab2e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800c3c0 tx=0x7f8ec800c880 comp rx=0 tx=0).stop 2023-11-02T08:36:31.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 shutdown_connections 2023-11-02T08:36:31.279 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ed80b4170 0x7f8ed81d9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ed80b13e0 0x7f8ed80abd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f8eb806bdc0 0x7f8eb806e280 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 --2- 172.21.15.151:0/2838159606 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed80a8df0 0x7f8ed80ab2e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 >> 172.21.15.151:0/2838159606 conn(0x7f8ed8065d80 msgr2=0x7f8ed8066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:31.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 shutdown_connections 2023-11-02T08:36:31.280 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 -- 172.21.15.151:0/2838159606 wait complete. 2023-11-02T08:36:31.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.273+0000 7f8edd4b6640 1 librados: shutdown 2023-11-02T08:36:31.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-02T08:36:31.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-02T08:36:31.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-02T08:36:31.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-02T08:36:31.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-02T08:36:31.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-02T08:36:31.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-02T08:36:31.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-02T08:36:31.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.464+0000 7f9d1596e640 10 monclient: get_monmap_and_config 2023-11-02T08:36:31.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.464+0000 7f9d1596e640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.464+0000 7f9d1596e640 10 monclient: monmap: 2023-11-02T08:36:31.469 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.470 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.470 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:31.465863+0000 2023-11-02T08:36:31.470 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:31.465863+0000 2023-11-02T08:36:31.470 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.471 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.471 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:31.471 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:31.471 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:31.471 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:31.472 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:31.472 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.464+0000 7f510a193640 10 monclient: get_monmap_and_config 2023-11-02T08:36:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.464+0000 7f510a193640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.464+0000 7f510a193640 10 monclient: monmap: 2023-11-02T08:36:31.473 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.474 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.474 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:31.465228+0000 2023-11-02T08:36:31.474 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:31.465228+0000 2023-11-02T08:36:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.475 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:31.476 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:31.476 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:31.476 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:31.477 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:31.477 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.479 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.466+0000 7f9d1596e640 1 Processor -- start 2023-11-02T08:36:31.479 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.466+0000 7f9d1596e640 1 -- start start 2023-11-02T08:36:31.480 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.466+0000 7f9d1596e640 10 monclient: init 2023-11-02T08:36:31.480 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.480 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-11-02T08:36:31.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9d1006b940 0x7f9d100ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient(hunting): picked mon.noname-c con 0x7f9d1006b940 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:31.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d10140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient(hunting): picked mon.noname-b con 0x7f9d100a8b30 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9d100a93d0 0x7f9d100a9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient(hunting): picked mon.noname-a con 0x7f9d100a93d0 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:31.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d100ab0d0 con 0x7f9d100a93d0 2023-11-02T08:36:31.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d100acbc0 con 0x7f9d1006b940 2023-11-02T08:36:31.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d100abbb0 con 0x7f9d100a8b30 2023-11-02T08:36:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.467+0000 7f9d1596e640 10 monclient(hunting): authenticate will time out at 1362.823472s 2023-11-02T08:36:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f9d0e7fc640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d10140bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f9d0e7fc640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d10140bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54666/0 (socket says 172.21.15.151:54666) 2023-11-02T08:36:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f9d0e7fc640 1 -- 172.21.15.151:0/2291996204 learned_addr learned my addr 172.21.15.151:0/2291996204 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f9d0e7fc640 10 monclient(hunting): get_auth_request con 0x7f9d100a8b30 auth_method 0 2023-11-02T08:36:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f9d0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f9d0e7fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f9d0e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f510a193640 1 Processor -- start 2023-11-02T08:36:31.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f510a193640 1 -- start start 2023-11-02T08:36:31.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.469+0000 7f510a193640 10 monclient: init 2023-11-02T08:36:31.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-11-02T08:36:31.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51040ab710 0x7f51040ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 10 monclient(hunting): picked mon.noname-b con 0x7f51040ab710 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51040ac590 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51041498a0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51040ac9d0 con 0x7f51040ac590 2023-11-02T08:36:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51040aa6e0 con 0x7f51041498a0 2023-11-02T08:36:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51040aa9c0 con 0x7f51040ab710 2023-11-02T08:36:31.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.470+0000 7f510a193640 10 monclient(hunting): authenticate will time out at 1362.826341s 2023-11-02T08:36:31.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.471+0000 7f9d0f7fe640 1 --2- 172.21.15.151:0/2291996204 >> v2:172.21.15.151:3300/0 conn(0x7f9d100a93d0 0x7f9d100a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.471+0000 7f7cd2ba2640 10 monclient: get_monmap_and_config 2023-11-02T08:36:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.471+0000 7f7cd2ba2640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.471+0000 7f7cd2ba2640 10 monclient: monmap: 2023-11-02T08:36:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:31.472349+0000 2023-11-02T08:36:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:31.472349+0000 2023-11-02T08:36:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.496 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:31.497 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:31.497 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:31.497 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:31.498 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:31.498 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.471+0000 7ff7e759e640 10 monclient: get_monmap_and_config 2023-11-02T08:36:31.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.471+0000 7ff7e759e640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.471+0000 7ff7e759e640 10 monclient: monmap: 2023-11-02T08:36:31.500 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.500 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.501 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:31.472836+0000 2023-11-02T08:36:31.501 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:31.472836+0000 2023-11-02T08:36:31.502 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.502 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.502 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:31.503 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:31.503 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:31.504 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:31.504 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:31.504 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.472+0000 7f7cd2ba2640 1 Processor -- start 2023-11-02T08:36:31.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.472+0000 7f9d0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.472+0000 7f9d0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.472+0000 7f9d0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.472+0000 7f7cd2ba2640 1 -- start start 2023-11-02T08:36:31.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.472+0000 7f7cd2ba2640 10 monclient: init 2023-11-02T08:36:31.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.472+0000 7f89ad026640 10 monclient: get_monmap_and_config 2023-11-02T08:36:31.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.472+0000 7f89ad026640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.472+0000 7f89ad026640 10 monclient: monmap: 2023-11-02T08:36:31.509 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.509 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.510 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:31.473717+0000 2023-11-02T08:36:31.510 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:31.473717+0000 2023-11-02T08:36:31.511 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.511 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.511 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:31.512 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:31.512 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:31.513 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:31.513 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:31.513 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f5109191640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51040ab710 0x7f51040ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-11-02T08:36:31.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc4034d70 0x7f7cc40c9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc4034d70 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7cc402ac60 0x7f7cc402b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f7cc402ac60 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:31.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7cc40cc2e0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:31.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc4007630 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc4035150 con 0x7f7cc402ac60 2023-11-02T08:36:31.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc402b690 con 0x7f7cc4034d70 2023-11-02T08:36:31.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f7cd2ba2640 10 monclient(hunting): authenticate will time out at 1362.829411s 2023-11-02T08:36:31.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f9f8e3ac640 10 monclient: get_monmap_and_config 2023-11-02T08:36:31.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f9f8e3ac640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f9f8e3ac640 10 monclient: monmap: 2023-11-02T08:36:31.520 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.521 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.521 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:31.474574+0000 2023-11-02T08:36:31.521 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:31.474574+0000 2023-11-02T08:36:31.521 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.522 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.522 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:31.522 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:31.522 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:31.523 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:31.523 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:31.523 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7ff7e759e640 1 Processor -- start 2023-11-02T08:36:31.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f89ad026640 1 Processor -- start 2023-11-02T08:36:31.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f89ad026640 1 -- start start 2023-11-02T08:36:31.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7f89ad026640 10 monclient: init 2023-11-02T08:36:31.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f5109191640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51040ab710 0x7f51040ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54670/0 (socket says 172.21.15.151:54670) 2023-11-02T08:36:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f5109191640 1 -- 172.21.15.151:0/632425280 learned_addr learned my addr 172.21.15.151:0/632425280 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f5109191640 10 monclient(hunting): get_auth_request con 0x7f51040ab710 auth_method 0 2023-11-02T08:36:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f5109191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f5109191640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f5109191640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f9f8e3ac640 1 Processor -- start 2023-11-02T08:36:31.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7ff7e759e640 1 -- start start 2023-11-02T08:36:31.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.473+0000 7ff7e759e640 10 monclient: init 2023-11-02T08:36:31.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f9f8e3ac640 1 -- start start 2023-11-02T08:36:31.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f89ad026640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f9f8e3ac640 10 monclient: init 2023-11-02T08:36:31.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f89ad026640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-11-02T08:36:31.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7f89ad026640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f89a80a9730 0x7f89a80711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7ff7e759e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.474+0000 7ff7e759e640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-11-02T08:36:31.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 10 monclient(hunting): picked mon.noname-a con 0x7f89a80a9730 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:31.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f89a813ea80 0x7f89a813eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 10 monclient(hunting): picked mon.noname-c con 0x7f89a813ea80 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:31.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7e806d280 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:31.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff7e80b1870 0x7ff7e8070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 10 monclient(hunting): picked mon.noname-a con 0x7ff7e80b1870 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:31.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff7e806d660 0x7ff7e80714a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 10 monclient(hunting): picked mon.noname-c con 0x7ff7e806d660 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:31.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e80719e0 con 0x7ff7e806d280 2023-11-02T08:36:31.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e806f260 con 0x7ff7e80b1870 2023-11-02T08:36:31.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e8145a60 con 0x7ff7e806d660 2023-11-02T08:36:31.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7ff7e759e640 10 monclient(hunting): authenticate will time out at 1362.830960s 2023-11-02T08:36:31.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89a8141c50 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:31.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a81417c0 con 0x7f89a8141c50 2023-11-02T08:36:31.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a80aa910 con 0x7f89a80a9730 2023-11-02T08:36:31.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a80a9b10 con 0x7f89a813ea80 2023-11-02T08:36:31.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f89ad026640 10 monclient(hunting): authenticate will time out at 1362.831281s 2023-11-02T08:36:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-11-02T08:36:31.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f80036e60 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:31.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9f800c2b30 0x7f9f800c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 10 monclient(hunting): picked mon.noname-c con 0x7f9f800c2b30 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:31.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f800c8f90 0x7f9f800c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 10 monclient(hunting): picked mon.noname-b con 0x7f9f800c8f90 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f7a3359e640 10 monclient: get_monmap_and_config 2023-11-02T08:36:31.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f7a3359e640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8002ea00 con 0x7f9f80036e60 2023-11-02T08:36:31.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f80037240 con 0x7f9f800c2b30 2023-11-02T08:36:31.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8002d5d0 con 0x7f9f800c8f90 2023-11-02T08:36:31.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f9f8e3ac640 10 monclient(hunting): authenticate will time out at 1362.831648s 2023-11-02T08:36:31.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.475+0000 7f7a3359e640 10 monclient: monmap: 2023-11-02T08:36:31.549 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.550 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.550 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:31.477023+0000 2023-11-02T08:36:31.550 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:31.477023+0000 2023-11-02T08:36:31.550 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.550 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.551 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:31.551 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:31.551 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:31.551 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:31.551 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:31.552 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2023-11-02T08:36:31.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 1 -- 172.21.15.151:0/2291996204 >> v2:172.21.15.178:3300/0 conn(0x7f9d1006b940 msgr2=0x7f9d100ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 1 --2- 172.21.15.151:0/2291996204 >> v2:172.21.15.178:3300/0 conn(0x7f9d1006b940 0x7f9d100ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 1 -- 172.21.15.151:0/2291996204 >> v2:172.21.15.151:3300/0 conn(0x7f9d100a93d0 msgr2=0x7f9d100a9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 1 --2- 172.21.15.151:0/2291996204 >> v2:172.21.15.151:3300/0 conn(0x7f9d100a93d0 0x7f9d100a9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 1 monclient: found mon.noname-b 2023-11-02T08:36:31.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f9d0e7fc640 1 -- 172.21.15.151:0/2291996204 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d10071400 con 0x7f9d100a8b30 2023-11-02T08:36:31.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f7a3359e640 1 Processor -- start 2023-11-02T08:36:31.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f7a3359e640 1 -- start start 2023-11-02T08:36:31.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.476+0000 7f7a3359e640 10 monclient: init 2023-11-02T08:36:31.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.477+0000 7f7a3359e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-11-02T08:36:31.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a2c037040 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:31.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a2c0c33b0 0x7f7a2c02cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 10 monclient(hunting): picked mon.noname-b con 0x7f7a2c0c33b0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7a2c0c8f60 0x7f7a2c0c9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 10 monclient(hunting): picked mon.noname-a con 0x7f7a2c0c8f60 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:31.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c02e490 con 0x7f7a2c037040 2023-11-02T08:36:31.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c037420 con 0x7f7a2c0c8f60 2023-11-02T08:36:31.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c02e6c0 con 0x7f7a2c0c33b0 2023-11-02T08:36:31.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.478+0000 7f7a3359e640 10 monclient(hunting): authenticate will time out at 1362.833982s 2023-11-02T08:36:31.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.479+0000 7f5109191640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.479+0000 7f5109191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.479+0000 7f5109191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.479+0000 7f9d0e7fc640 1 --2- 172.21.15.151:0/2291996204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d10140bf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9d10071f40 tx=0x7f9d0402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba94f05e262b0d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.482+0000 7f5109191640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2023-11-02T08:36:31.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.482+0000 7f5109191640 1 -- 172.21.15.151:0/632425280 >> v1:172.21.15.178:6789/0 conn(0x7f51041498a0 legacy=0x7f5104150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.482+0000 7f5109191640 1 -- 172.21.15.151:0/632425280 >> v1:172.21.15.151:6789/0 conn(0x7f51040ac590 legacy=0x7f5104149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.482+0000 7f5109191640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.482+0000 7f5109191640 1 monclient: found mon.noname-b 2023-11-02T08:36:31.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.482+0000 7f5109191640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.482+0000 7f5109191640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.482+0000 7f5109191640 1 -- 172.21.15.151:0/632425280 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51040aaca0 con 0x7f51040ab710 2023-11-02T08:36:31.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.485+0000 7f9f8dbab640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f800c8f90 0x7f9f800c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.486+0000 7f9f8dbab640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f800c8f90 0x7f9f800c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54678/0 (socket says 172.21.15.151:54678) 2023-11-02T08:36:31.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.486+0000 7f9f8dbab640 1 -- 172.21.15.151:0/2520099337 learned_addr learned my addr 172.21.15.151:0/2520099337 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.486+0000 7f9f8dbab640 10 monclient(hunting): get_auth_request con 0x7f9f800c8f90 auth_method 0 2023-11-02T08:36:31.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.486+0000 7f9f8dbab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.486+0000 7f9f8dbab640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.486+0000 7f9f8dbab640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.486+0000 7f7cd1ba0640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc4034d70 0x7f7cc40c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.487+0000 7f5109191640 1 --2- 172.21.15.151:0/632425280 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51040ab710 0x7f51040ac050 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f50f40098a0 tx=0x7f50f402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8263936ba53af0dd server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.488+0000 7f9f8dbab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.488+0000 7f9f8dbab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.488+0000 7f9f8dbab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f89a659c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f89a813ea80 0x7f89a813eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f9f8dbab640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2023-11-02T08:36:31.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f9f8dbab640 1 -- 172.21.15.151:0/2520099337 >> v2:172.21.15.178:3300/0 conn(0x7f9f800c2b30 msgr2=0x7f9f800c2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f9f8dbab640 1 --2- 172.21.15.151:0/2520099337 >> v2:172.21.15.178:3300/0 conn(0x7f9f800c2b30 0x7f9f800c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f9f8dbab640 1 -- 172.21.15.151:0/2520099337 >> v1:172.21.15.178:6789/0 conn(0x7f9f80036e60 legacy=0x7f9f8002cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f9f8dbab640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f9f8dbab640 1 monclient: found mon.noname-b 2023-11-02T08:36:31.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f9f8dbab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f9f8dbab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.489+0000 7f9f8dbab640 1 -- 172.21.15.151:0/2520099337 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800c34e0 con 0x7f9f800c8f90 2023-11-02T08:36:31.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.491+0000 7f89a659c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f89a813ea80 0x7f89a813eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36170/0 (socket says 172.21.15.151:36170) 2023-11-02T08:36:31.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.491+0000 7f89a659c640 1 -- 172.21.15.151:0/1700195696 learned_addr learned my addr 172.21.15.151:0/1700195696 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.491+0000 7f89a659c640 10 monclient(hunting): get_auth_request con 0x7f89a813ea80 auth_method 0 2023-11-02T08:36:31.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.491+0000 7f89a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.491+0000 7f89a659c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.491+0000 7f89a659c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.491+0000 7ff7e6d9d640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff7e806d660 0x7ff7e80714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.492+0000 7f89a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.492+0000 7f89a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.492+0000 7f89a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.493+0000 7f7a3259c640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f7a2c037040 0x7f7a2c0c2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:42878/0 (socket says 172.21.15.151:42878) 2023-11-02T08:36:31.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.493+0000 7f7a3259c640 1 -- 172.21.15.151:0/932382172 learned_addr learned my addr 172.21.15.151:0/932382172 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:31.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.494+0000 7f795f539640 10 monclient: get_monmap_and_config 2023-11-02T08:36:31.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.494+0000 7f795f539640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.494+0000 7f795f539640 10 monclient: monmap: 2023-11-02T08:36:31.582 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.582 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.583 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:31.495368+0000 2023-11-02T08:36:31.583 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:31.495368+0000 2023-11-02T08:36:31.583 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.583 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.583 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:31.584 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:31.584 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:31.584 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:31.584 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:31.584 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.494+0000 7f795f539640 1 Processor -- start 2023-11-02T08:36:31.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 1 -- start start 2023-11-02T08:36:31.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient: init 2023-11-02T08:36:31.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-11-02T08:36:31.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f79580b3850 0x7f79580b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient(hunting): picked mon.noname-a con 0x7f79580b3850 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:31.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79580ae8e0 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:31.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f79580b0950 0x7f79580b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient(hunting): picked mon.noname-c con 0x7f79580b0950 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:31.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795813f570 con 0x7f79580ae8e0 2023-11-02T08:36:31.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7958141900 con 0x7f79580b3850 2023-11-02T08:36:31.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7958142800 con 0x7f79580b0950 2023-11-02T08:36:31.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.495+0000 7f795f539640 10 monclient(hunting): authenticate will time out at 1362.851776s 2023-11-02T08:36:31.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7ff7e6d9d640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff7e806d660 0x7ff7e80714a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36168/0 (socket says 172.21.15.151:36168) 2023-11-02T08:36:31.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7ff7e6d9d640 1 -- 172.21.15.151:0/3603486526 learned_addr learned my addr 172.21.15.151:0/3603486526 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7ff7e6d9d640 10 monclient(hunting): get_auth_request con 0x7ff7e806d660 auth_method 0 2023-11-02T08:36:31.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7ff7e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7ff7e6d9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7ff7e6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7f89a659c640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2023-11-02T08:36:31.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7f89a659c640 1 -- 172.21.15.151:0/1700195696 >> v2:172.21.15.151:3300/0 conn(0x7f89a80a9730 msgr2=0x7f89a80711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7f89a659c640 1 --2- 172.21.15.151:0/1700195696 >> v2:172.21.15.151:3300/0 conn(0x7f89a80a9730 0x7f89a80711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7f89a659c640 1 -- 172.21.15.151:0/1700195696 >> v1:172.21.15.151:6789/0 conn(0x7f89a8141c50 legacy=0x7f89a81420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7f89a659c640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7f89a659c640 1 monclient: found mon.noname-c 2023-11-02T08:36:31.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7f89a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7f89a659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.496+0000 7f89a659c640 1 -- 172.21.15.151:0/1700195696 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a813f400 con 0x7f89a813ea80 2023-11-02T08:36:31.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.498+0000 7ff7e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.498+0000 7ff7e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.498+0000 7ff7e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.501+0000 7ff7e6d9d640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2023-11-02T08:36:31.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.501+0000 7ff7e6d9d640 1 -- 172.21.15.151:0/3603486526 >> v2:172.21.15.151:3300/0 conn(0x7ff7e80b1870 msgr2=0x7ff7e8070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.501+0000 7ff7e6d9d640 1 --2- 172.21.15.151:0/3603486526 >> v2:172.21.15.151:3300/0 conn(0x7ff7e80b1870 0x7ff7e8070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.501+0000 7ff7e6d9d640 1 -- 172.21.15.151:0/3603486526 >> v1:172.21.15.178:6789/0 conn(0x7ff7e806d280 legacy=0x7ff7e8145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.501+0000 7ff7e6d9d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.501+0000 7ff7e6d9d640 1 monclient: found mon.noname-c 2023-11-02T08:36:31.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.501+0000 7ff7e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.501+0000 7ff7e6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.501+0000 7ff7e6d9d640 1 -- 172.21.15.151:0/3603486526 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e806f4b0 con 0x7ff7e806d660 2023-11-02T08:36:31.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.506+0000 7ff7e6d9d640 1 --2- 172.21.15.151:0/3603486526 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e806d660 0x7ff7e80714a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0009e50 tx=0x7ff7e002be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df9ca5accc12e0de server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.507+0000 7f89a659c640 1 --2- 172.21.15.151:0/1700195696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 0x7f89a813eec0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f89a80724a0 tx=0x7f899c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c80910b976311fcc server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.508+0000 7f7cd23a1640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f7cc40cc2e0 0x7f7cc40cc770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:42850/0 (socket says 172.21.15.151:42850) 2023-11-02T08:36:31.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.508+0000 7f7cd23a1640 1 -- 172.21.15.151:0/4169367311 learned_addr learned my addr 172.21.15.151:0/4169367311 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:31.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89a5d9b640 1 -- 172.21.15.151:0/1700195696 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c0317e0 con 0x7f89a813ea80 2023-11-02T08:36:31.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89a5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:31.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89a5d9b640 10 monclient: dump: 2023-11-02T08:36:31.604 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.604 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.605 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.605 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.605 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.606 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.606 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.606 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.607 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.607 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89a5d9b640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89a5d9b640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 5 monclient: authenticate success, global_id 4303 2023-11-02T08:36:31.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:31.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89a5d9b640 1 -- 172.21.15.151:0/1700195696 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899c031940 con 0x7f89a813ea80 2023-11-02T08:36:31.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89a5d9b640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:31.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 4 set_mon_vals no callback set 2023-11-02T08:36:31.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 10 monclient: shutdown 2023-11-02T08:36:31.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:31.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 1 -- 172.21.15.151:0/1700195696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 msgr2=0x7f89a813eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 1 --2- 172.21.15.151:0/1700195696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 0x7f89a813eec0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f89a80724a0 tx=0x7f899c02b880 comp rx=0 tx=0).stop 2023-11-02T08:36:31.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 1 -- 172.21.15.151:0/1700195696 shutdown_connections 2023-11-02T08:36:31.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 1 --2- 172.21.15.151:0/1700195696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 0x7f89a813eec0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.613 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 1 --2- 172.21.15.151:0/1700195696 >> v2:172.21.15.151:3300/0 conn(0x7f89a80a9730 0x7f89a80711e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.514+0000 7f89ad026640 1 -- 172.21.15.151:0/1700195696 >> 172.21.15.151:0/1700195696 conn(0x7f89a8069e50 msgr2=0x7f89a806a230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:31.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.515+0000 7f89ad026640 1 -- 172.21.15.151:0/1700195696 shutdown_connections 2023-11-02T08:36:31.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.515+0000 7f89ad026640 1 -- 172.21.15.151:0/1700195696 wait complete. 2023-11-02T08:36:31.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.515+0000 7f89ad026640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.516+0000 7f89ad026640 10 monclient: monmap: 2023-11-02T08:36:31.615 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.616 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:31.616 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:31.617 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:31.617 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.617 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.618 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:31.618 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:31.618 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:31.619 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.516+0000 7f89ad026640 1 librados: starting msgr at 2023-11-02T08:36:31.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.516+0000 7f89ad026640 1 librados: starting objecter 2023-11-02T08:36:31.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.516+0000 7f89ad026640 1 Processor -- start 2023-11-02T08:36:31.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.516+0000 7f89ad026640 1 -- start start 2023-11-02T08:36:31.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.516+0000 7f89ad026640 1 librados: setting wanted keys 2023-11-02T08:36:31.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.516+0000 7f89ad026640 1 librados: calling monclient init 2023-11-02T08:36:31.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.516+0000 7f89ad026640 10 monclient: init 2023-11-02T08:36:31.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:36:31.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a80a9730 0x7f89a811a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient(hunting): picked mon.noname-a con 0x7f89a80a9730 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:31.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 0x7f89a80abff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient(hunting): picked mon.noname-b con 0x7f89a813ea80 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:31.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f89a8141c50 0x7f89a81d9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient(hunting): picked mon.noname-c con 0x7f89a8141c50 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a811aed0 con 0x7f89a80a9730 2023-11-02T08:36:31.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a80aca00 con 0x7f89a813ea80 2023-11-02T08:36:31.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.626 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a8144730 con 0x7f89a8141c50 2023-11-02T08:36:31.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f89ad026640 10 monclient(hunting): authenticate will time out at 1362.873050s 2023-11-02T08:36:31.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f795dad6640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f79580b0950 0x7f79580b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f795dad6640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f79580b0950 0x7f79580b0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36188/0 (socket says 172.21.15.151:36188) 2023-11-02T08:36:31.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f795dad6640 1 -- 172.21.15.151:0/4132371579 learned_addr learned my addr 172.21.15.151:0/4132371579 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f795dad6640 10 monclient(hunting): get_auth_request con 0x7f79580b0950 auth_method 0 2023-11-02T08:36:31.630 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f795dad6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.630 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f795dad6640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.630 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.517+0000 7f795dad6640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.518+0000 7f795dad6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.518+0000 7f795dad6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.518+0000 7f795dad6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 1 -- 172.21.15.151:0/3603486526 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e002c9f0 con 0x7ff7e806d660 2023-11-02T08:36:31.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:31.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 10 monclient: dump: 2023-11-02T08:36:31.638 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.638 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.639 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.639 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.639 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.640 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.640 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.640 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.641 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.641 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.641 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 1 -- 172.21.15.151:0/3603486526 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e002cb50 con 0x7ff7e806d660 2023-11-02T08:36:31.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 1 -- 172.21.15.151:0/3603486526 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e00343f0 con 0x7ff7e806d660 2023-11-02T08:36:31.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e759e640 5 monclient: authenticate success, global_id 4306 2023-11-02T08:36:31.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e759e640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:31.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e759e640 4 set_mon_vals no callback set 2023-11-02T08:36:31.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:31.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.519+0000 7ff7e559a640 10 monclient: dump: 2023-11-02T08:36:31.645 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.646 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.646 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.646 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.647 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.647 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.647 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.648 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.648 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.648 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 10 monclient: shutdown 2023-11-02T08:36:31.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:31.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 1 -- 172.21.15.151:0/3603486526 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e806d660 msgr2=0x7ff7e80714a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 1 --2- 172.21.15.151:0/3603486526 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e806d660 0x7ff7e80714a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0009e50 tx=0x7ff7e002be50 comp rx=0 tx=0).stop 2023-11-02T08:36:31.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 1 -- 172.21.15.151:0/3603486526 shutdown_connections 2023-11-02T08:36:31.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 1 --2- 172.21.15.151:0/3603486526 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e806d660 0x7ff7e80714a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 1 --2- 172.21.15.151:0/3603486526 >> v2:172.21.15.151:3300/0 conn(0x7ff7e80b1870 0x7ff7e8070f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 1 -- 172.21.15.151:0/3603486526 >> 172.21.15.151:0/3603486526 conn(0x7ff7e80b3970 msgr2=0x7ff7e80b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:31.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 1 -- 172.21.15.151:0/3603486526 shutdown_connections 2023-11-02T08:36:31.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7ff7e759e640 1 -- 172.21.15.151:0/3603486526 wait complete. 2023-11-02T08:36:31.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7f795dad6640 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2023-11-02T08:36:31.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7f795dad6640 1 -- 172.21.15.151:0/4132371579 >> v2:172.21.15.151:3300/0 conn(0x7f79580b3850 msgr2=0x7f79580b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7f795dad6640 1 --2- 172.21.15.151:0/4132371579 >> v2:172.21.15.151:3300/0 conn(0x7f79580b3850 0x7f79580b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7f795dad6640 1 -- 172.21.15.151:0/4132371579 >> v1:172.21.15.151:6789/0 conn(0x7f79580ae8e0 legacy=0x7f795813ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7f795dad6640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7f795dad6640 1 monclient: found mon.noname-c 2023-11-02T08:36:31.654 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7f795dad6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7f795dad6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.520+0000 7f795dad6640 1 -- 172.21.15.151:0/4132371579 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79580aed50 con 0x7f79580b0950 2023-11-02T08:36:31.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.521+0000 7ff7e759e640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.521+0000 7ff7e759e640 10 monclient: monmap: 2023-11-02T08:36:31.658 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.658 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:31.659 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:31.659 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:31.659 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.660 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.660 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:31.660 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:31.661 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:31.661 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.521+0000 7ff7e759e640 1 librados: starting msgr at 2023-11-02T08:36:31.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.521+0000 7ff7e759e640 1 librados: starting objecter 2023-11-02T08:36:31.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.521+0000 7ff7e759e640 1 Processor -- start 2023-11-02T08:36:31.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 1 -- start start 2023-11-02T08:36:31.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 1 librados: setting wanted keys 2023-11-02T08:36:31.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 1 librados: calling monclient init 2023-11-02T08:36:31.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient: init 2023-11-02T08:36:31.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:36:31.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e806d280 0x7ff7e8140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient(hunting): picked mon.noname-a con 0x7ff7e806d280 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:31.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7e806d660 0x7ff7e81429e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient(hunting): picked mon.noname-c con 0x7ff7e806d660 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e80b1870 0x7ff7e8137820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient(hunting): picked mon.noname-b con 0x7ff7e80b1870 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:31.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e8141080 con 0x7ff7e806d280 2023-11-02T08:36:31.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80b7dc0 con 0x7ff7e80b1870 2023-11-02T08:36:31.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e8141510 con 0x7ff7e806d660 2023-11-02T08:36:31.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.522+0000 7ff7e759e640 10 monclient(hunting): authenticate will time out at 1362.878456s 2023-11-02T08:36:31.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.523+0000 7f795dad6640 1 --2- 172.21.15.151:0/4132371579 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79580b0950 0x7f79580b0d60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79580b2d10 tx=0x7f795002be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6650ec79fc687a76 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.523+0000 7f89a659c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 0x7f89a80abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.524+0000 7f89a659c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 0x7f89a80abff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36198/0 (socket says 172.21.15.151:36198) 2023-11-02T08:36:31.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.524+0000 7f89a659c640 1 -- 172.21.15.151:0/3968162331 learned_addr learned my addr 172.21.15.151:0/3968162331 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.524+0000 7f89a659c640 10 monclient(hunting): get_auth_request con 0x7f89a813ea80 auth_method 0 2023-11-02T08:36:31.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.524+0000 7f89a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.524+0000 7f89a659c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.524+0000 7f89a659c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.527+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258649193 0 0) 0x7f7a2c02e490 con 0x7f7a2c037040 2023-11-02T08:36:31.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.527+0000 7f7a3159a640 10 monclient(hunting): _negotiate 2023-11-02T08:36:31.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.527+0000 7f7a3159a640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.527+0000 7f7a3159a640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.527+0000 7f7a3159a640 10 monclient(hunting): my global_id is 4312 2023-11-02T08:36:31.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.527+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a1c003520 con 0x7f7a2c037040 2023-11-02T08:36:31.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.528+0000 7f89a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.528+0000 7f89a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.528+0000 7f89a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.531+0000 7f7cd0b9e640 1 -- 172.21.15.151:0/4169367311 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968044138 0 0) 0x7f7cc4007630 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.532+0000 7ff7e6d9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e80b1870 0x7ff7e8137820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.532+0000 7f7cd0b9e640 10 monclient(hunting): _negotiate 2023-11-02T08:36:31.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.532+0000 7f7cd0b9e640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.532+0000 7f7cd0b9e640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.532+0000 7f7cd0b9e640 10 monclient(hunting): my global_id is 4315 2023-11-02T08:36:31.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.532+0000 7f7cd0b9e640 1 -- 172.21.15.151:0/4169367311 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb8003520 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.536+0000 7f89a6d9d640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a80a9730 0x7f89a811a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.537+0000 7f7cd1ba0640 10 monclient(hunting): get_auth_request con 0x7f7cc4034d70 auth_method 0 2023-11-02T08:36:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.537+0000 7f7cd1ba0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.537+0000 7f7cd1ba0640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.537+0000 7f7cd1ba0640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.537+0000 7f89a6d9d640 10 monclient(hunting): get_auth_request con 0x7f89a80a9730 auth_method 0 2023-11-02T08:36:31.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.537+0000 7f89a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.537+0000 7f89a6d9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.537+0000 7f89a6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.539+0000 7f89a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.539+0000 7f89a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.539+0000 7f89a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2023-11-02T08:36:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 1 -- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f89a8141c50 msgr2=0x7f89a81d9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f89a8141c50 0x7f89a81d9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 1 -- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a80a9730 msgr2=0x7f89a811a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f89a80a9730 0x7f89a811a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 1 monclient: found mon.noname-b 2023-11-02T08:36:31.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.540+0000 7f89a659c640 1 -- 172.21.15.151:0/3968162331 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c0094d0 con 0x7f89a813ea80 2023-11-02T08:36:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.541+0000 7ff7e6d9d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e80b1870 0x7ff7e8137820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36212/0 (socket says 172.21.15.151:36212) 2023-11-02T08:36:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.541+0000 7ff7e6d9d640 1 -- 172.21.15.151:0/4278818941 learned_addr learned my addr 172.21.15.151:0/4278818941 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.541+0000 7ff7e6d9d640 10 monclient(hunting): get_auth_request con 0x7ff7e80b1870 auth_method 0 2023-11-02T08:36:31.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.541+0000 7ff7e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.541+0000 7ff7e6d9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.541+0000 7ff7e6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.541+0000 7f7cd1ba0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.541+0000 7f7cd1ba0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.541+0000 7f7cd1ba0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 1 -- 172.21.15.151:0/4132371579 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795002c9f0 con 0x7f79580b0950 2023-11-02T08:36:31.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:31.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 10 monclient: dump: 2023-11-02T08:36:31.685 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.685 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.686 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.686 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.686 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.686 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.686 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.687 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.687 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.687 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 1 -- 172.21.15.151:0/4132371579 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795002cb50 con 0x7f79580b0950 2023-11-02T08:36:31.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 1 -- 172.21.15.151:0/4132371579 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79500343f0 con 0x7f79580b0950 2023-11-02T08:36:31.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f795f539640 5 monclient: authenticate success, global_id 4309 2023-11-02T08:36:31.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f795f539640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:31.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f795f539640 4 set_mon_vals no callback set 2023-11-02T08:36:31.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:31.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.543+0000 7f794ffff640 10 monclient: dump: 2023-11-02T08:36:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.690 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.691 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.691 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.691 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.691 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.692 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 10 monclient: shutdown 2023-11-02T08:36:31.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:31.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 1 -- 172.21.15.151:0/4132371579 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79580b0950 msgr2=0x7f79580b0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 1 --2- 172.21.15.151:0/4132371579 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79580b0950 0x7f79580b0d60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79580b2d10 tx=0x7f795002be50 comp rx=0 tx=0).stop 2023-11-02T08:36:31.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 1 -- 172.21.15.151:0/4132371579 shutdown_connections 2023-11-02T08:36:31.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 1 --2- 172.21.15.151:0/4132371579 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79580b0950 0x7f79580b0d60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 1 --2- 172.21.15.151:0/4132371579 >> v2:172.21.15.151:3300/0 conn(0x7f79580b3850 0x7f79580b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 1 -- 172.21.15.151:0/4132371579 >> 172.21.15.151:0/4132371579 conn(0x7f7958065960 msgr2=0x7f7958065d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:31.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 1 -- 172.21.15.151:0/4132371579 shutdown_connections 2023-11-02T08:36:31.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f795f539640 1 -- 172.21.15.151:0/4132371579 wait complete. 2023-11-02T08:36:31.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7ff7e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7ff7e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7ff7e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f7cd1ba0640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2023-11-02T08:36:31.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f7cd1ba0640 1 -- 172.21.15.151:0/4169367311 >> v2:172.21.15.151:3300/0 conn(0x7f7cc402ac60 msgr2=0x7f7cc402b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f7cd1ba0640 1 --2- 172.21.15.151:0/4169367311 >> v2:172.21.15.151:3300/0 conn(0x7f7cc402ac60 0x7f7cc402b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f7cd1ba0640 1 -- 172.21.15.151:0/4169367311 >> v1:172.21.15.178:6789/0 conn(0x7f7cc40cc2e0 legacy=0x7f7cc40cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f795f539640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.697 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f795f539640 10 monclient: monmap: 2023-11-02T08:36:31.697 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.697 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:31.697 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:31.698 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:31.698 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.698 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.698 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:31.698 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:31.699 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:31.699 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f7cd1ba0640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f7cd1ba0640 1 monclient: found mon.noname-b 2023-11-02T08:36:31.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f7cd1ba0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f7cd1ba0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.544+0000 7f7cd1ba0640 1 -- 172.21.15.151:0/4169367311 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc40cce80 con 0x7f7cc4034d70 2023-11-02T08:36:31.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f795f539640 1 librados: starting msgr at 2023-11-02T08:36:31.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f795f539640 1 librados: starting objecter 2023-11-02T08:36:31.701 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3133370089 0 0) 0x7f7a1c003520 con 0x7f7a2c037040 2023-11-02T08:36:31.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a2c0c33b0 msgr2=0x7f7a2c02cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 --2- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a2c0c33b0 0x7f7a2c02cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 >> v2:172.21.15.151:3300/0 conn(0x7f7a2c0c8f60 msgr2=0x7f7a2c0c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 --2- 172.21.15.151:0/932382172 >> v2:172.21.15.151:3300/0 conn(0x7f7a2c0c8f60 0x7f7a2c0c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 monclient: found mon.noname-c-legacy 2023-11-02T08:36:31.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.178:6789/0 2023-11-02T08:36:31.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c02e9a0 con 0x7f7a2c037040 2023-11-02T08:36:31.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 650167734 0 0) 0x7f7a24001670 con 0x7f7a2c037040 2023-11-02T08:36:31.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-11-02T08:36:31.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient: dump: 2023-11-02T08:36:31.705 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.705 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.705 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.705 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.706 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.706 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.706 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.706 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.706 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.707 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] but i'm connected to v1:172.21.15.178:6789/0, reconnecting 2023-11-02T08:36:31.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 >> v1:172.21.15.178:6789/0 conn(0x7f7a2c037040 legacy=0x7f7a2c0c2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f795f539640 1 Processor -- start 2023-11-02T08:36:31.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:36:31.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 --2- 172.21.15.151:0/932382172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a1c003970 0x7f7a1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f795f539640 1 -- start start 2023-11-02T08:36:31.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f795f539640 1 librados: setting wanted keys 2023-11-02T08:36:31.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f795f539640 1 librados: calling monclient init 2023-11-02T08:36:31.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f795f539640 10 monclient: init 2023-11-02T08:36:31.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f89a659c640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 0x7f89a80abff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f899c0090b0 tx=0x7f899c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f898f7fe640 10 client.?.objecter ms_handle_connect 0x7f89a813ea80 2023-11-02T08:36:31.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f898f7fe640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:31.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient(hunting): picked mon.b con 0x7f7a1c003970 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:31.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 --2- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a1c0047c0 0x7f7a1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient(hunting): picked mon.a con 0x7f7a1c0047c0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:31.711 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 --2- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a2c0c33b0 0x7f7a1c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient(hunting): picked mon.c con 0x7f7a2c0c33b0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c037420 con 0x7f7a1c0047c0 2023-11-02T08:36:31.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c02e6c0 con 0x7f7a1c003970 2023-11-02T08:36:31.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c005670 con 0x7f7a2c0c33b0 2023-11-02T08:36:31.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.545+0000 7f7a3159a640 10 monclient(hunting): _finish_auth 0 2023-11-02T08:36:31.714 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-02T08:36:31.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79580ae8e0 0x7f7958140ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient(hunting): picked mon.noname-c con 0x7f79580ae8e0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79580b0950 0x7f79581449b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.715 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient(hunting): picked mon.noname-a con 0x7f79580b0950 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:31.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79580b3850 0x7f79581d9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient(hunting): picked mon.noname-b con 0x7f79580b3850 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:31.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7958145350 con 0x7f79580b0950 2023-11-02T08:36:31.716 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79581457e0 con 0x7f79580b3850 2023-11-02T08:36:31.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79581413b0 con 0x7f79580ae8e0 2023-11-02T08:36:31.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.717 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.546+0000 7f795f539640 10 monclient(hunting): authenticate will time out at 1362.902464s 2023-11-02T08:36:31.718 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.547+0000 7f9f8dbab640 1 --2- 172.21.15.151:0/2520099337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f800c8f90 0x7f9f800c93e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9f88066b90 tx=0x7f9f880a56c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deaa383977fde70 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2023-11-02T08:36:31.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 1 -- 172.21.15.151:0/4278818941 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7e806d660 msgr2=0x7ff7e81429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7e806d660 0x7ff7e81429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 1 -- 172.21.15.151:0/4278818941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e806d280 msgr2=0x7ff7e8140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.719 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e806d280 0x7ff7e8140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 1 monclient: found mon.noname-b 2023-11-02T08:36:31.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.548+0000 7ff7e6d9d640 1 -- 172.21.15.151:0/4278818941 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e0009b30 con 0x7ff7e80b1870 2023-11-02T08:36:31.721 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.550+0000 7f7cd1ba0640 1 --2- 172.21.15.151:0/4169367311 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc4034d70 0x7f7cc40c9610 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc0097a0 tx=0x7f7cbc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46bf000bb03d34fd server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.551+0000 7f795cad4640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79580b0950 0x7f79581449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.552+0000 7f795cad4640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79580b0950 0x7f79581449b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38510/0 (socket says 172.21.15.151:38510) 2023-11-02T08:36:31.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.552+0000 7f795cad4640 1 -- 172.21.15.151:0/2818850046 learned_addr learned my addr 172.21.15.151:0/2818850046 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.552+0000 7f795cad4640 10 monclient(hunting): get_auth_request con 0x7f79580b0950 auth_method 0 2023-11-02T08:36:31.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.552+0000 7f795cad4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.552+0000 7f795cad4640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.552+0000 7f795cad4640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.552+0000 7f7a31d9b640 1 --2- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a1c0047c0 0x7f7a1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.553+0000 7f7a31d9b640 10 monclient(hunting): get_auth_request con 0x7f7a1c0047c0 auth_method 0 2023-11-02T08:36:31.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.553+0000 7f7a31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.553+0000 7f7a31d9b640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.553+0000 7f7a31d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:36:31.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.553+0000 7f795cad4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.553+0000 7f795cad4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.553+0000 7f795cad4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.554+0000 7f7a31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.554+0000 7f7a31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.554+0000 7f7a31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-02T08:36:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2023-11-02T08:36:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 1 -- 172.21.15.151:0/2818850046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79580ae8e0 msgr2=0x7f7958140ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79580ae8e0 0x7f7958140ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 1 -- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79580b3850 msgr2=0x7f79581d9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79580b3850 0x7f79581d9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 1 monclient: found mon.noname-a 2023-11-02T08:36:31.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:31.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f795cad4640 1 -- 172.21.15.151:0/2818850046 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7950009b30 con 0x7f79580b0950 2023-11-02T08:36:31.733 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 1 -- 172.21.15.151:0/632425280 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f40317c0 con 0x7f51040ab710 2023-11-02T08:36:31.733 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:31.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 10 monclient: dump: 2023-11-02T08:36:31.734 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.735 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.735 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.735 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.736 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.736 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.737 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.737 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.737 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.738 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f510a193640 5 monclient: authenticate success, global_id 4241 2023-11-02T08:36:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f510a193640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 1 -- 172.21.15.151:0/632425280 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f4031920 con 0x7f51040ab710 2023-11-02T08:36:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f510a193640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:31.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f510a193640 4 set_mon_vals no callback set 2023-11-02T08:36:31.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 1 -- 172.21.15.151:0/632425280 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f40317c0 con 0x7f51040ab710 2023-11-02T08:36:31.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:31.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.557+0000 7f50fbfff640 10 monclient: dump: 2023-11-02T08:36:31.741 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.741 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.741 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.741 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.742 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.742 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.742 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.742 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.742 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.743 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.558+0000 7f510a193640 10 monclient: shutdown 2023-11-02T08:36:31.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.558+0000 7f510a193640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:31.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.558+0000 7f510a193640 1 -- 172.21.15.151:0/632425280 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51040ab710 msgr2=0x7f51040ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.743 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.558+0000 7f510a193640 1 --2- 172.21.15.151:0/632425280 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51040ab710 0x7f51040ac050 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f50f40098a0 tx=0x7f50f402b860 comp rx=0 tx=0).stop 2023-11-02T08:36:31.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.558+0000 7f510a193640 1 -- 172.21.15.151:0/632425280 shutdown_connections 2023-11-02T08:36:31.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.558+0000 7f510a193640 1 --2- 172.21.15.151:0/632425280 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51040ab710 0x7f51040ac050 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.558+0000 7f510a193640 1 -- 172.21.15.151:0/632425280 >> 172.21.15.151:0/632425280 conn(0x7f51040af560 msgr2=0x7f51040af970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:31.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.558+0000 7f510a193640 1 -- 172.21.15.151:0/632425280 shutdown_connections 2023-11-02T08:36:31.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.558+0000 7f510a193640 1 -- 172.21.15.151:0/632425280 wait complete. 2023-11-02T08:36:31.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.559+0000 7f510a193640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.559+0000 7f510a193640 10 monclient: monmap: 2023-11-02T08:36:31.745 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.745 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:31.745 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:31.745 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:31.746 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.746 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.746 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:31.746 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:31.746 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:31.747 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.559+0000 7f510a193640 1 librados: starting msgr at 2023-11-02T08:36:31.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.559+0000 7f510a193640 1 librados: starting objecter 2023-11-02T08:36:31.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 1 Processor -- start 2023-11-02T08:36:31.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 1 -- start start 2023-11-02T08:36:31.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 1 librados: setting wanted keys 2023-11-02T08:36:31.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 1 librados: calling monclient init 2023-11-02T08:36:31.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 10 monclient: init 2023-11-02T08:36:31.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 1 -- 172.21.15.151:0/3968162331 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c032d70 con 0x7f89a813ea80 2023-11-02T08:36:31.749 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:31.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 10 monclient: dump: 2023-11-02T08:36:31.750 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.750 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.750 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.751 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.751 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.751 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.751 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.751 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.752 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.752 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 1 -- 172.21.15.151:0/3968162331 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899c0453f0 con 0x7f89a813ea80 2023-11-02T08:36:31.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89ad026640 5 monclient: authenticate success, global_id 4318 2023-11-02T08:36:31.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89ad026640 10 monclient: _renew_subs 2023-11-02T08:36:31.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89ad026640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a81449b0 con 0x7f89a813ea80 2023-11-02T08:36:31.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89ad026640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89ad026640 10 monclient: _renew_subs 2023-11-02T08:36:31.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89ad026640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 1 -- 172.21.15.151:0/3968162331 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c0457a0 con 0x7f89a813ea80 2023-11-02T08:36:31.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a81da450 con 0x7f89a813ea80 2023-11-02T08:36:31.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89ad026640 1 librados: init done 2023-11-02T08:36:31.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f89a559a640 4 set_mon_vals no callback set 2023-11-02T08:36:31.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:31.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f898f7fe640 10 monclient: dump: 2023-11-02T08:36:31.756 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.756 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.756 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.756 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.757 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.757 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.757 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.757 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.757 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.757 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:36:31.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51040ab710 0x7f510414fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 10 monclient(hunting): picked mon.noname-b con 0x7f51040ab710 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:31.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.560+0000 7f510a193640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51040ac590 0x7f5104138e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 10 monclient(hunting): picked mon.noname-a con 0x7f51040ac590 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:31.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51041498a0 0x7f5104139380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 10 monclient(hunting): picked mon.noname-c con 0x7f51041498a0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5104151b70 con 0x7f51040ac590 2023-11-02T08:36:31.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5104150690 con 0x7f51040ab710 2023-11-02T08:36:31.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510414e4b0 con 0x7f51041498a0 2023-11-02T08:36:31.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f510a193640 10 monclient(hunting): authenticate will time out at 1362.916910s 2023-11-02T08:36:31.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.562+0000 7f89ad026640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:31.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.562+0000 7f89ad026640 15 client.4318.objecter _session_command_op_assign -1 1 2023-11-02T08:36:31.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.562+0000 7f89ad026640 20 client.4318.objecter _get_session osd=-1 returning homeless 2023-11-02T08:36:31.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.562+0000 7f89ad026640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 10 monclient(hunting): handle_auth_done global_id 4312 payload 293 2023-11-02T08:36:31.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 1 -- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a2c0c33b0 msgr2=0x7f7a1c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 1 --2- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a2c0c33b0 0x7f7a1c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 1 -- 172.21.15.151:0/932382172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a1c003970 msgr2=0x7f7a1c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 1 --2- 172.21.15.151:0/932382172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a1c003970 0x7f7a1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 1 monclient: found mon.a 2023-11-02T08:36:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 1 -- 172.21.15.151:0/932382172 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c003520 con 0x7f7a1c0047c0 2023-11-02T08:36:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a3359e640 5 monclient: authenticate success, global_id 4312 2023-11-02T08:36:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.561+0000 7f7a3359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.562+0000 7f89ad026640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.562+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f89a80721a0 con 0x7f89a813ea80 2023-11-02T08:36:31.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 1 -- 172.21.15.151:0/2291996204 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04031870 con 0x7f9d100a8b30 2023-11-02T08:36:31.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:31.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 10 monclient: dump: 2023-11-02T08:36:31.769 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.769 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.770 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.770 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.770 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.770 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.770 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.771 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.771 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.771 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d1596e640 5 monclient: authenticate success, global_id 4238 2023-11-02T08:36:31.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d1596e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:31.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 1 -- 172.21.15.151:0/2291996204 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d040319d0 con 0x7f9d100a8b30 2023-11-02T08:36:31.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d1596e640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:31.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d1596e640 4 set_mon_vals no callback set 2023-11-02T08:36:31.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 1 -- 172.21.15.151:0/2291996204 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04031d60 con 0x7f9d100a8b30 2023-11-02T08:36:31.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:31.773 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.563+0000 7f9d0dffb640 10 monclient: dump: 2023-11-02T08:36:31.774 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.774 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.774 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.774 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.774 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.775 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.775 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.775 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.775 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.775 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 10 monclient: shutdown 2023-11-02T08:36:31.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:31.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 1 -- 172.21.15.151:0/2291996204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 msgr2=0x7f9d10140bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 1 --2- 172.21.15.151:0/2291996204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d10140bf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9d10071f40 tx=0x7f9d0402b910 comp rx=0 tx=0).stop 2023-11-02T08:36:31.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 1 -- 172.21.15.151:0/2291996204 shutdown_connections 2023-11-02T08:36:31.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 1 --2- 172.21.15.151:0/2291996204 >> v2:172.21.15.151:3300/0 conn(0x7f9d100a93d0 0x7f9d100a9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 1 --2- 172.21.15.151:0/2291996204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d10140bf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 1 --2- 172.21.15.151:0/2291996204 >> v2:172.21.15.178:3300/0 conn(0x7f9d1006b940 0x7f9d100ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 1 -- 172.21.15.151:0/2291996204 >> 172.21.15.151:0/2291996204 conn(0x7f9d100b2ad0 msgr2=0x7f9d100b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:31.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 1 -- 172.21.15.151:0/2291996204 shutdown_connections 2023-11-02T08:36:31.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.564+0000 7f9d1596e640 1 -- 172.21.15.151:0/2291996204 wait complete. 2023-11-02T08:36:31.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.565+0000 7f9d1596e640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.565+0000 7f9d1596e640 10 monclient: monmap: 2023-11-02T08:36:31.778 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.778 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:31.779 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:31.779 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:31.779 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.779 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.779 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:31.780 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:31.780 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:31.780 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.565+0000 7f9d1596e640 1 librados: starting msgr at 2023-11-02T08:36:31.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.565+0000 7f9d1596e640 1 librados: starting objecter 2023-11-02T08:36:31.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.565+0000 7f9d1596e640 1 Processor -- start 2023-11-02T08:36:31.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.565+0000 7f9d1596e640 1 -- start start 2023-11-02T08:36:31.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.565+0000 7f9d1596e640 1 librados: setting wanted keys 2023-11-02T08:36:31.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.565+0000 7f9d1596e640 1 librados: calling monclient init 2023-11-02T08:36:31.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.565+0000 7f9d1596e640 10 monclient: init 2023-11-02T08:36:31.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.782 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:36:31.782 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d1006b940 0x7f9d10133c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.782 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient(hunting): picked mon.noname-a con 0x7f9d1006b940 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:31.782 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d1006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.782 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient(hunting): picked mon.noname-c con 0x7f9d100a8b30 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d100a93d0 0x7f9d1006fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient(hunting): picked mon.noname-b con 0x7f9d100a93d0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:31.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10070300 con 0x7f9d1006b940 2023-11-02T08:36:31.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10139c00 con 0x7f9d100a93d0 2023-11-02T08:36:31.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1006f8a0 con 0x7f9d100a8b30 2023-11-02T08:36:31.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f9d1596e640 10 monclient(hunting): authenticate will time out at 1362.922295s 2023-11-02T08:36:31.785 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f795cad4640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79580b0950 0x7f79581449b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f794800c430 tx=0x7f794800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.785 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f794dffb640 10 client.?.objecter ms_handle_connect 0x7f79580b0950 2023-11-02T08:36:31.785 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.566+0000 7f794dffb640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:31.786 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f7a31d9b640 1 --2- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a1c0047c0 0x7f7a1c004c40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009860 tx=0x7f7a1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d2458eaa62bae5 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 1 -- 172.21.15.151:0/2520099337 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f880aa5d0 con 0x7f9f800c8f90 2023-11-02T08:36:31.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:31.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 10 monclient: dump: 2023-11-02T08:36:31.789 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.789 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.789 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.789 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.789 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.790 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.790 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.790 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.790 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.790 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 1 -- 172.21.15.151:0/2520099337 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f880a8050 con 0x7f9f800c8f90 2023-11-02T08:36:31.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f8e3ac640 5 monclient: authenticate success, global_id 4244 2023-11-02T08:36:31.791 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f8e3ac640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:31.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f8e3ac640 4 set_mon_vals no callback set 2023-11-02T08:36:31.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 1 -- 172.21.15.151:0/2520099337 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f880aa9f0 con 0x7f9f800c8f90 2023-11-02T08:36:31.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.792 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:31.793 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.567+0000 7f9f7ffff640 10 monclient: dump: 2023-11-02T08:36:31.793 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.793 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.793 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.793 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.794 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.794 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.794 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.794 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.794 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.795 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 10 monclient: shutdown 2023-11-02T08:36:31.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:31.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2520099337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f800c8f90 msgr2=0x7f9f800c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.795 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 1 --2- 172.21.15.151:0/2520099337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f800c8f90 0x7f9f800c93e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9f88066b90 tx=0x7f9f880a56c0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2520099337 shutdown_connections 2023-11-02T08:36:31.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 1 --2- 172.21.15.151:0/2520099337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f800c8f90 0x7f9f800c93e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 1 --2- 172.21.15.151:0/2520099337 >> v2:172.21.15.178:3300/0 conn(0x7f9f800c2b30 0x7f9f800c2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2520099337 >> 172.21.15.151:0/2520099337 conn(0x7f9f800062b0 msgr2=0x7f9f800066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:31.796 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2520099337 shutdown_connections 2023-11-02T08:36:31.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.568+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2520099337 wait complete. 2023-11-02T08:36:31.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f9f8e3ac640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.797 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f9f8e3ac640 10 monclient: monmap: 2023-11-02T08:36:31.797 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.797 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:31.798 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:31.798 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:31.798 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.798 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.798 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:31.799 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:31.799 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:31.799 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f9f8e3ac640 1 librados: starting msgr at 2023-11-02T08:36:31.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f9f8e3ac640 1 librados: starting objecter 2023-11-02T08:36:31.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f9f8e3ac640 1 Processor -- start 2023-11-02T08:36:31.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f9f8e3ac640 1 -- start start 2023-11-02T08:36:31.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f9f8e3ac640 1 librados: setting wanted keys 2023-11-02T08:36:31.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f9f8e3ac640 1 librados: calling monclient init 2023-11-02T08:36:31.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f9f8e3ac640 10 monclient: init 2023-11-02T08:36:31.801 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd0b9e640 1 -- 172.21.15.151:0/4169367311 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc031780 con 0x7f7cc4034d70 2023-11-02T08:36:31.802 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd0b9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.802 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd0b9e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:31.802 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd0b9e640 10 monclient: dump: 2023-11-02T08:36:31.802 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.803 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.803 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.803 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.803 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.803 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.804 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.804 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.804 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.804 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd0b9e640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.804 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd0b9e640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd0b9e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd2ba2640 5 monclient: authenticate success, global_id 4247 2023-11-02T08:36:31.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd2ba2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:31.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd0b9e640 1 -- 172.21.15.151:0/4169367311 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cbc0318e0 con 0x7f7cc4034d70 2023-11-02T08:36:31.805 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd0b9e640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd2ba2640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:31.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.569+0000 7f7cd2ba2640 4 set_mon_vals no callback set 2023-11-02T08:36:31.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd0b9e640 1 -- 172.21.15.151:0/4169367311 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc031c70 con 0x7f7cc4034d70 2023-11-02T08:36:31.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd0b9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.806 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd0b9e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:31.807 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd0b9e640 10 monclient: dump: 2023-11-02T08:36:31.807 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.807 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.807 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.807 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.808 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.808 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.808 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.808 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.808 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.809 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 10 monclient: shutdown 2023-11-02T08:36:31.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:31.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/4169367311 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc4034d70 msgr2=0x7f7cc40c9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.809 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 1 --2- 172.21.15.151:0/4169367311 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc4034d70 0x7f7cc40c9610 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc0097a0 tx=0x7f7cbc02b820 comp rx=0 tx=0).stop 2023-11-02T08:36:31.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/4169367311 shutdown_connections 2023-11-02T08:36:31.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 1 --2- 172.21.15.151:0/4169367311 >> v2:172.21.15.151:3300/0 conn(0x7f7cc402ac60 0x7f7cc402b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 1 --2- 172.21.15.151:0/4169367311 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc4034d70 0x7f7cc40c9610 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.810 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/4169367311 >> 172.21.15.151:0/4169367311 conn(0x7f7cc40061c0 msgr2=0x7f7cc40065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:31.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f9f8e3ac640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f9f8e3ac640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-02T08:36:31.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f9f8e3ac640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f80036e60 0x7f9f800c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.811 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/4169367311 shutdown_connections 2023-11-02T08:36:31.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f9f8e3ac640 10 monclient(hunting): picked mon.noname-c con 0x7f9f80036e60 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f800c2b30 0x7f9f8015dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.812 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.570+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/4169367311 wait complete. 2023-11-02T08:36:31.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 10 monclient(hunting): picked mon.noname-a con 0x7f9f800c2b30 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:31.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f800c8f90 0x7f9f800c3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 10 monclient(hunting): picked mon.noname-b con 0x7f9f800c8f90 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:31.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015dbc0 con 0x7f9f800c2b30 2023-11-02T08:36:31.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015e520 con 0x7f9f800c8f90 2023-11-02T08:36:31.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800c3cc0 con 0x7f9f80036e60 2023-11-02T08:36:31.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.815 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f9f8e3ac640 10 monclient(hunting): authenticate will time out at 1362.927021s 2023-11-02T08:36:31.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f7cd2ba2640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.816 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f7cd2ba2640 10 monclient: monmap: 2023-11-02T08:36:31.817 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.817 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:31.817 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:31.817 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:31.817 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.818 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.818 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:31.818 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:31.818 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:31.818 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.818 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f7cd2ba2640 1 librados: starting msgr at 2023-11-02T08:36:31.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f7cd2ba2640 1 librados: starting objecter 2023-11-02T08:36:31.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.571+0000 7f7cd2ba2640 1 Processor -- start 2023-11-02T08:36:31.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 1 -- start start 2023-11-02T08:36:31.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 1 librados: setting wanted keys 2023-11-02T08:36:31.819 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 1 librados: calling monclient init 2023-11-02T08:36:31.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient: init 2023-11-02T08:36:31.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:36:31.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cc402ac60 0x7f7cc40c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc402ac60 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:31.821 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cc4034d70 0x7f7cc40c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.821 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f7cc4034d70 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:31.821 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc40cc2e0 0x7f7cc4037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.821 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f7cc40cc2e0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.821 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40c2ec0 con 0x7f7cc4034d70 2023-11-02T08:36:31.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40c43d0 con 0x7f7cc402ac60 2023-11-02T08:36:31.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40c30c0 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.822 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.823 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.572+0000 7f7cd2ba2640 10 monclient(hunting): authenticate will time out at 1362.928502s 2023-11-02T08:36:31.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.573+0000 7f5109992640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51041498a0 0x7f5104139380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.574+0000 7f5108990640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51040ac590 0x7f5104138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5109992640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51041498a0 0x7f5104139380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54754/0 (socket says 172.21.15.151:54754) 2023-11-02T08:36:31.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5109992640 1 -- 172.21.15.151:0/4225271198 learned_addr learned my addr 172.21.15.151:0/4225271198 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5109992640 10 monclient(hunting): get_auth_request con 0x7f51041498a0 auth_method 0 2023-11-02T08:36:31.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5109992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5109992640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5109992640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5108990640 10 monclient(hunting): get_auth_request con 0x7f51040ac590 auth_method 0 2023-11-02T08:36:31.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5108990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5108990640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.575+0000 7f5108990640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 1 -- 172.21.15.151:0/2818850046 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948011790 con 0x7f79580b0950 2023-11-02T08:36:31.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.828 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:31.828 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 10 monclient: dump: 2023-11-02T08:36:31.828 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.828 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.829 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.829 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.829 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.829 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.829 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.830 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 1 -- 172.21.15.151:0/2818850046 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79480118f0 con 0x7f79580b0950 2023-11-02T08:36:31.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 1 -- 172.21.15.151:0/2818850046 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948011ce0 con 0x7f79580b0950 2023-11-02T08:36:31.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794dffb640 10 monclient: dump: 2023-11-02T08:36:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.832 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.833 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.834 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.834 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.834 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f794f7fe640 4 set_mon_vals no callback set 2023-11-02T08:36:31.834 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f795f539640 5 monclient: authenticate success, global_id 4272 2023-11-02T08:36:31.834 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f795f539640 10 monclient: _renew_subs 2023-11-02T08:36:31.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f795f539640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:31.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7958144ef0 con 0x7f79580b0950 2023-11-02T08:36:31.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f795f539640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f795f539640 10 monclient: _renew_subs 2023-11-02T08:36:31.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f795f539640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:31.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79581da2c0 con 0x7f79580b0950 2023-11-02T08:36:31.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.576+0000 7f795f539640 1 librados: init done 2023-11-02T08:36:31.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f5108990640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f5108990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f5108990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f5109992640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f5109992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f5109992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.838 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f795f539640 10 client.4272.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:31.838 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f9d0e7fc640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d1006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f795f539640 15 client.4272.objecter _session_command_op_assign -1 1 2023-11-02T08:36:31.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f795f539640 20 client.4272.objecter _get_session osd=-1 returning homeless 2023-11-02T08:36:31.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f795f539640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f795f539640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:31.840 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.577+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7958145490 con 0x7f79580b0950 2023-11-02T08:36:31.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f7cd139f640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cc4034d70 0x7f7cc40c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f9d0e7fc640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d1006f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54766/0 (socket says 172.21.15.151:54766) 2023-11-02T08:36:31.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f9d0e7fc640 1 -- 172.21.15.151:0/3515774077 learned_addr learned my addr 172.21.15.151:0/3515774077 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f9d0e7fc640 10 monclient(hunting): get_auth_request con 0x7f9d100a8b30 auth_method 0 2023-11-02T08:36:31.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f9d0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f9d0e7fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f9d0e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f7cd139f640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cc4034d70 0x7f7cc40c3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38562/0 (socket says 172.21.15.151:38562) 2023-11-02T08:36:31.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f7cd139f640 1 -- 172.21.15.151:0/141595971 learned_addr learned my addr 172.21.15.151:0/141595971 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f7cd139f640 10 monclient(hunting): get_auth_request con 0x7f7cc4034d70 auth_method 0 2023-11-02T08:36:31.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f7cd139f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f7cd139f640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.579+0000 7f7cd139f640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.580+0000 7f7cd23a1640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc40cc2e0 0x7f7cc4037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.845 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28032600 con 0x7f7a1c0047c0 2023-11-02T08:36:31.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:31.846 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 10 monclient: dump: 2023-11-02T08:36:31.846 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.847 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.847 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.847 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.847 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.847 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.847 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.848 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.848 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.848 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.848 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2802e050 con 0x7f7a1c0047c0 2023-11-02T08:36:31.848 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.849 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 1 -- 172.21.15.151:0/932382172 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a280329c0 con 0x7f7a1c0047c0 2023-11-02T08:36:31.849 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3359e640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:31.849 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3359e640 4 set_mon_vals no callback set 2023-11-02T08:36:31.849 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.849 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:31.850 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f7a3159a640 10 monclient: dump: 2023-11-02T08:36:31.850 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.850 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.850 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.850 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.851 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.851 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.851 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.851 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.851 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.852 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.852 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f9d0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.852 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f9d0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.852 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.581+0000 7f9d0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7a3359e640 10 monclient: shutdown 2023-11-02T08:36:31.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7a3359e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:31.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7a3359e640 1 -- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a1c0047c0 msgr2=0x7f7a1c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7a3359e640 1 --2- 172.21.15.151:0/932382172 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a1c0047c0 0x7f7a1c004c40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009860 tx=0x7f7a1c0057f0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.854 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7a38bc9640 1 -- 172.21.15.151:0/932382172 reap_dead start 2023-11-02T08:36:31.854 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7cd23a1640 10 monclient(hunting): get_auth_request con 0x7f7cc40cc2e0 auth_method 0 2023-11-02T08:36:31.854 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7cd23a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.854 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7cd23a1640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7cd23a1640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7a3359e640 1 -- 172.21.15.151:0/932382172 shutdown_connections 2023-11-02T08:36:31.855 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7a3359e640 1 -- 172.21.15.151:0/932382172 >> 172.21.15.151:0/932382172 conn(0x7f7a2c006330 msgr2=0x7f7a2c006740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:31.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7a3359e640 1 -- 172.21.15.151:0/932382172 shutdown_connections 2023-11-02T08:36:31.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f9f8d3aa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f80036e60 0x7f9f800c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.856 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7a3359e640 1 -- 172.21.15.151:0/932382172 wait complete. 2023-11-02T08:36:31.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7cd23a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7cd23a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.857 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.582+0000 7f7cd23a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.583+0000 7f7a3359e640 10 monclient: build_initial_monmap 2023-11-02T08:36:31.859 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.583+0000 7f7a3359e640 10 monclient: monmap: 2023-11-02T08:36:31.860 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:31.860 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:31.860 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:31.860 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:31.860 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:31.861 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.861 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:31.861 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:31.861 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:31.861 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.583+0000 7f7a3359e640 1 librados: starting msgr at 2023-11-02T08:36:31.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.583+0000 7f7a3359e640 1 librados: starting objecter 2023-11-02T08:36:31.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.583+0000 7f7a3359e640 1 Processor -- start 2023-11-02T08:36:31.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.583+0000 7f7a3359e640 1 -- start start 2023-11-02T08:36:31.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.583+0000 7f7a3359e640 1 librados: setting wanted keys 2023-11-02T08:36:31.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.583+0000 7f7a3359e640 1 librados: calling monclient init 2023-11-02T08:36:31.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.583+0000 7f7a3359e640 10 monclient: init 2023-11-02T08:36:31.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:31.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:36:31.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a1c0047c0 0x7f7a2c0089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient(hunting): picked mon.noname-a con 0x7f7a1c0047c0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:31.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a2c0c51c0 0x7f7a2c02fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient(hunting): picked mon.noname-b con 0x7f7a2c0c51c0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:31.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a2c0c5c30 0x7f7a2c0c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient(hunting): picked mon.noname-c con 0x7f7a2c0c5c30 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:31.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0c56f0 con 0x7f7a1c0047c0 2023-11-02T08:36:31.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c030130 con 0x7f7a2c0c51c0 2023-11-02T08:36:31.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:31.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c009640 con 0x7f7a2c0c5c30 2023-11-02T08:36:31.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:31.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.584+0000 7f7a3359e640 10 monclient(hunting): authenticate will time out at 1362.940529s 2023-11-02T08:36:31.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2023-11-02T08:36:31.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 1 -- 172.21.15.151:0/3515774077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d100a93d0 msgr2=0x7f9d1006fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d100a93d0 0x7f9d1006fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 1 -- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d1006b940 msgr2=0x7f9d10133c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d1006b940 0x7f9d10133c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 1 monclient: found mon.noname-c 2023-11-02T08:36:31.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.585+0000 7f9d0e7fc640 1 -- 172.21.15.151:0/3515774077 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d040094d0 con 0x7f9d100a8b30 2023-11-02T08:36:31.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-11-02T08:36:31.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 1 -- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cc402ac60 msgr2=0x7f7cc40c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cc402ac60 0x7f7cc40c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 1 -- 172.21.15.151:0/141595971 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cc4034d70 msgr2=0x7f7cc40c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7cc4034d70 0x7f7cc40c3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 1 monclient: found mon.noname-c 2023-11-02T08:36:31.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.588+0000 7f7cd23a1640 1 -- 172.21.15.151:0/141595971 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc009400 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.590+0000 7f9f8d3aa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f80036e60 0x7f9f800c8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54768/0 (socket says 172.21.15.151:54768) 2023-11-02T08:36:31.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.590+0000 7f9f8d3aa640 1 -- 172.21.15.151:0/2642604601 learned_addr learned my addr 172.21.15.151:0/2642604601 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.590+0000 7f9f8d3aa640 10 monclient(hunting): get_auth_request con 0x7f9f80036e60 auth_method 0 2023-11-02T08:36:31.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.590+0000 7f9f8d3aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.590+0000 7f9f8d3aa640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.590+0000 7f9f8d3aa640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2023-11-02T08:36:31.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 1 -- 172.21.15.151:0/4225271198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51041498a0 msgr2=0x7f5104139380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51041498a0 0x7f5104139380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 1 -- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51040ab710 msgr2=0x7f510414fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51040ab710 0x7f510414fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 1 monclient: found mon.noname-a 2023-11-02T08:36:31.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:31.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.592+0000 7f5108990640 1 -- 172.21.15.151:0/4225271198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f40094d0 con 0x7f51040ac590 2023-11-02T08:36:31.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.593+0000 7f9f8d3aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.593+0000 7f9f8d3aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.593+0000 7f9f8d3aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.593+0000 7f7cd23a1640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc40cc2e0 0x7f7cc4037fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc02c840 tx=0x7f7ccc067960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.593+0000 7f7cc27fc640 10 client.?.objecter ms_handle_connect 0x7f7cc40cc2e0 2023-11-02T08:36:31.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.593+0000 7f7cc27fc640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:31.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.594+0000 7f9d0e7fc640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d1006f280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9d0402b950 tx=0x7f9d040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.594+0000 7f9cf7fff640 10 client.?.objecter ms_handle_connect 0x7f9d100a8b30 2023-11-02T08:36:31.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.594+0000 7f9cf7fff640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:31.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.595+0000 7f9f8cba9640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f800c2b30 0x7f9f8015dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.595+0000 7f9f8cba9640 10 monclient(hunting): get_auth_request con 0x7f9f800c2b30 auth_method 0 2023-11-02T08:36:31.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.595+0000 7f9f8cba9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.595+0000 7f9f8cba9640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.595+0000 7f9f8cba9640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.598+0000 7f5108990640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51040ac590 0x7f5104138e40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f510000c410 tx=0x7f510000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.598+0000 7f50f9ffb640 10 client.?.objecter ms_handle_connect 0x7f51040ac590 2023-11-02T08:36:31.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.598+0000 7f50f9ffb640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:31.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.598+0000 7f9f8cba9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.598+0000 7f9f8cba9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8cba9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2023-11-02T08:36:31.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 1 -- 172.21.15.151:0/2642604601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f800c8f90 msgr2=0x7f9f800c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f800c8f90 0x7f9f800c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 1 -- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f800c2b30 msgr2=0x7f9f8015dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9f800c2b30 0x7f9f8015dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 1 monclient: found mon.noname-c 2023-11-02T08:36:31.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.599+0000 7f9f8d3aa640 1 -- 172.21.15.151:0/2642604601 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800c4920 con 0x7f9f80036e60 2023-11-02T08:36:31.885 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.600+0000 7f898f7fe640 1 -- 172.21.15.151:0/3968162331 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f899c04d3f0 con 0x7f89a813ea80 2023-11-02T08:36:31.885 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.600+0000 7f898f7fe640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f899406bdc0 0x7f899406e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.885 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.600+0000 7f89a6d9d640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f899406bdc0 0x7f899406e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.600+0000 7f89a6d9d640 10 monclient: get_auth_request con 0x7f899406bdc0 auth_method 0 2023-11-02T08:36:31.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.601+0000 7f898f7fe640 10 client.4318.objecter ms_handle_connect 0x7f899406bdc0 2023-11-02T08:36:31.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.601+0000 7f89a6d9d640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f899406bdc0 0x7f899406e280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f89a000a400 tx=0x7f89a000d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 1 -- 172.21.15.151:0/3968162331 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f899c07a470 con 0x7f89a813ea80 2023-11-02T08:36:31.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 10 client.4318.objecter ms_dispatch 0x7f89a8000b90 osd_map(21..21 src has 1..21) v4 2023-11-02T08:36:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 3 client.4318.objecter handle_osd_map got epochs [21,21] > 0 2023-11-02T08:36:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 3 client.4318.objecter handle_osd_map decoding full epoch 21 2023-11-02T08:36:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 10 client.4318.objecter checking command 1 2023-11-02T08:36:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:6816/86100,v1:172.21.15.151:6817/86100] conn(0x7f89940721a0 0x7f89940745c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 20 client.4318.objecter _get_session s=0x7f8994071ad0 osd=2 3 2023-11-02T08:36:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 20 client.4318.objecter put_session s=0x7f8994071ad0 osd=2 3 2023-11-02T08:36:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 15 client.4318.objecter _session_command_op_remove -1 1 2023-11-02T08:36:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 20 client.4318.objecter _get_session s=0x7f8994071ad0 osd=2 3 2023-11-02T08:36:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 20 client.4318.objecter get_session s=0x7f8994071ad0 osd=2 3 2023-11-02T08:36:31.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 15 client.4318.objecter _session_command_op_assign 2 1 2023-11-02T08:36:31.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 20 client.4318.objecter put_session s=0x7f8994071ad0 osd=2 4 2023-11-02T08:36:31.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 10 client.4318.objecter _send_command 1 2023-11-02T08:36:31.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 1 -- 172.21.15.151:0/3968162331 --> [v2:172.21.15.151:6816/86100,v1:172.21.15.151:6817/86100] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8994074c70 con 0x7f89940721a0 2023-11-02T08:36:31.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.607+0000 7f898f7fe640 20 client.4318.objecter dump_active .. 0 homeless 2023-11-02T08:36:31.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.608+0000 7f89a759e640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:6816/86100,v1:172.21.15.151:6817/86100] conn(0x7f89940721a0 0x7f89940745c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.608+0000 7f89a759e640 10 monclient: get_auth_request con 0x7f89940721a0 auth_method 0 2023-11-02T08:36:31.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.610+0000 7f9f8d3aa640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f80036e60 0x7f9f800c8350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800c410 tx=0x7f9f7800c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.610+0000 7f9f7dffb640 10 client.?.objecter ms_handle_connect 0x7f9f80036e60 2023-11-02T08:36:31.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.610+0000 7f9f7dffb640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.612+0000 7f7a31d9b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a2c0c51c0 0x7f7a2c02fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.612+0000 7ff7e6d9d640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e80b1870 0x7ff7e8137820 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0036040 tx=0x7ff7e0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.612+0000 7ff7d6ffd640 10 client.?.objecter ms_handle_connect 0x7ff7e80b1870 2023-11-02T08:36:31.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.612+0000 7ff7d6ffd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:31.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.612+0000 7f89a759e640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:6816/86100,v1:172.21.15.151:6817/86100] conn(0x7f89940721a0 0x7f89940745c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.612+0000 7f898f7fe640 10 client.4318.objecter ms_handle_connect 0x7f89940721a0 2023-11-02T08:36:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.613+0000 7f7a31d9b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a2c0c51c0 0x7f7a2c02fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36256/0 (socket says 172.21.15.151:36256) 2023-11-02T08:36:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.613+0000 7f7a31d9b640 1 -- 172.21.15.151:0/3709476426 learned_addr learned my addr 172.21.15.151:0/3709476426 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.613+0000 7f7a31d9b640 10 monclient(hunting): get_auth_request con 0x7f7a2c0c51c0 auth_method 0 2023-11-02T08:36:31.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.613+0000 7f7a31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:31.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.613+0000 7f7a31d9b640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:31.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.613+0000 7f7a31d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:31.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.614+0000 7f794dffb640 1 -- 172.21.15.151:0/2818850046 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f794801c460 con 0x7f79580b0950 2023-11-02T08:36:31.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.615+0000 7f794dffb640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f793c06be10 0x7f793c06e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.615+0000 7f795d2d5640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f793c06be10 0x7f793c06e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.615+0000 7f795d2d5640 10 monclient: get_auth_request con 0x7f793c06be10 auth_method 0 2023-11-02T08:36:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.615+0000 7f7a31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.615+0000 7f7a31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:31.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.615+0000 7f7a31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:31.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f795d2d5640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f793c06be10 0x7f793c06e2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7954009670 tx=0x7f7954006bf0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f794dffb640 10 client.4272.objecter ms_handle_connect 0x7f793c06be10 2023-11-02T08:36:31.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 1 -- 172.21.15.151:0/141595971 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc066c30 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:31.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 10 monclient: dump: 2023-11-02T08:36:31.898 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.898 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.899 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.899 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.899 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.899 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.899 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.900 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.900 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.900 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 1 -- 172.21.15.151:0/141595971 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc066db0 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 1 -- 172.21.15.151:0/141595971 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc0671e0 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.902 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:31.902 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc27fc640 10 monclient: dump: 2023-11-02T08:36:31.902 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.902 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.902 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.903 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.903 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.903 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.903 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.903 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.903 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.904 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cc3fff640 4 set_mon_vals no callback set 2023-11-02T08:36:31.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cd2ba2640 5 monclient: authenticate success, global_id 4256 2023-11-02T08:36:31.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cd2ba2640 10 monclient: _renew_subs 2023-11-02T08:36:31.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cd2ba2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc40c33a0 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cd2ba2640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cd2ba2640 10 monclient: _renew_subs 2023-11-02T08:36:31.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cd2ba2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc40c25b0 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.616+0000 7f7cd2ba2640 1 librados: init done 2023-11-02T08:36:31.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 1 -- 172.21.15.151:0/3515774077 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04031840 con 0x7f9d100a8b30 2023-11-02T08:36:31.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:31.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 10 monclient: dump: 2023-11-02T08:36:31.907 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.907 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.907 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.908 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.908 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.908 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.908 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.908 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.908 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.909 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 1 -- 172.21.15.151:0/3515774077 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d040443f0 con 0x7f9d100a8b30 2023-11-02T08:36:31.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 1 -- 172.21.15.151:0/3515774077 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d040447a0 con 0x7f9d100a8b30 2023-11-02T08:36:31.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:31.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9cf7fff640 10 monclient: dump: 2023-11-02T08:36:31.911 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.911 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.911 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.911 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.911 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.912 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.912 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.912 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.912 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.912 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d1596e640 5 monclient: authenticate success, global_id 4253 2023-11-02T08:36:31.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d1596e640 10 monclient: _renew_subs 2023-11-02T08:36:31.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d1596e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1006e130 con 0x7f9d100a8b30 2023-11-02T08:36:31.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d1596e640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d1596e640 10 monclient: _renew_subs 2023-11-02T08:36:31.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d1596e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1012f920 con 0x7f9d100a8b30 2023-11-02T08:36:31.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d1596e640 1 librados: init done 2023-11-02T08:36:31.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f7cd2ba2640 10 client.4256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:31.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f7cd2ba2640 15 client.4256.objecter _session_command_op_assign -1 1 2023-11-02T08:36:31.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f9d0d7fa640 4 set_mon_vals no callback set 2023-11-02T08:36:31.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f7cd2ba2640 20 client.4256.objecter _get_session osd=-1 returning homeless 2023-11-02T08:36:31.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f7cd2ba2640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f7cd2ba2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.617+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7cc40d6990 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.618+0000 7f9d1596e640 10 client.4253.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:31.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.618+0000 7f9d1596e640 15 client.4253.objecter _session_command_op_assign -1 1 2023-11-02T08:36:31.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.618+0000 7f9d1596e640 20 client.4253.objecter _get_session osd=-1 returning homeless 2023-11-02T08:36:31.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.618+0000 7f9d1596e640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.618+0000 7f9d1596e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.618+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9cf00028a0 con 0x7f9d100a8b30 2023-11-02T08:36:31.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2023-11-02T08:36:31.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 1 -- 172.21.15.151:0/3709476426 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a2c0c5c30 msgr2=0x7f7a2c0c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a2c0c5c30 0x7f7a2c0c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 1 -- 172.21.15.151:0/3709476426 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a1c0047c0 msgr2=0x7f7a2c0089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:31.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a1c0047c0 0x7f7a2c0089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:31.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 10 monclient: _finish_hunting 0 2023-11-02T08:36:31.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 1 monclient: found mon.noname-b 2023-11-02T08:36:31.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:31.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.621+0000 7f7a31d9b640 1 -- 172.21.15.151:0/3709476426 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c0c6770 con 0x7f7a2c0c51c0 2023-11-02T08:36:31.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 1 -- 172.21.15.151:0/2818850046 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7948011e40 con 0x7f79580b0950 2023-11-02T08:36:31.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 10 client.4272.objecter ms_dispatch 0x7f7958000b90 osd_map(21..21 src has 1..21) v4 2023-11-02T08:36:31.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 3 client.4272.objecter handle_osd_map got epochs [21,21] > 0 2023-11-02T08:36:31.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 3 client.4272.objecter handle_osd_map decoding full epoch 21 2023-11-02T08:36:31.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 10 client.4272.objecter checking command 1 2023-11-02T08:36:31.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6802/86123,v1:172.21.15.178:6803/86123] conn(0x7f793c0721f0 0x7f793c074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 20 client.4272.objecter _get_session s=0x7f793c071b20 osd=7 3 2023-11-02T08:36:31.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 20 client.4272.objecter put_session s=0x7f793c071b20 osd=7 3 2023-11-02T08:36:31.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 15 client.4272.objecter _session_command_op_remove -1 1 2023-11-02T08:36:31.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 20 client.4272.objecter _get_session s=0x7f793c071b20 osd=7 3 2023-11-02T08:36:31.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 20 client.4272.objecter get_session s=0x7f793c071b20 osd=7 3 2023-11-02T08:36:31.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 15 client.4272.objecter _session_command_op_assign 7 1 2023-11-02T08:36:31.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 20 client.4272.objecter put_session s=0x7f793c071b20 osd=7 4 2023-11-02T08:36:31.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 10 client.4272.objecter _send_command 1 2023-11-02T08:36:31.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 1 -- 172.21.15.151:0/2818850046 --> [v2:172.21.15.178:6802/86123,v1:172.21.15.178:6803/86123] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f793c074cc0 con 0x7f793c0721f0 2023-11-02T08:36:31.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 20 client.4272.objecter dump_active .. 0 homeless 2023-11-02T08:36:31.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 1 -- 172.21.15.151:0/2818850046 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7948054850 con 0x7f79580b0950 2023-11-02T08:36:31.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.624+0000 7f794dffb640 10 monclient: handle_get_version_reply finishing 1 version 21 2023-11-02T08:36:31.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.625+0000 7f795dad6640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6802/86123,v1:172.21.15.178:6803/86123] conn(0x7f793c0721f0 0x7f793c074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.626+0000 7f795dad6640 10 monclient: get_auth_request con 0x7f793c0721f0 auth_method 0 2023-11-02T08:36:31.927 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.627+0000 7ff7d6ffd640 1 -- 172.21.15.151:0/4278818941 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e003e450 con 0x7ff7e80b1870 2023-11-02T08:36:31.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:31.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 10 monclient: dump: 2023-11-02T08:36:31.928 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.928 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.929 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.929 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.929 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.929 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.929 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.930 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.930 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.930 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.930 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.930 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.931 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.931 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 1 -- 172.21.15.151:0/4278818941 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e003e5b0 con 0x7ff7e80b1870 2023-11-02T08:36:31.931 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.931 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 1 -- 172.21.15.151:0/4278818941 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e003e960 con 0x7ff7e80b1870 2023-11-02T08:36:31.931 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.932 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:31.932 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7d6ffd640 10 monclient: dump: 2023-11-02T08:36:31.932 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.932 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.932 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.933 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.933 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.933 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.933 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.933 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.934 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.934 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e4d99640 4 set_mon_vals no callback set 2023-11-02T08:36:31.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e759e640 5 monclient: authenticate success, global_id 4321 2023-11-02T08:36:31.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e759e640 10 monclient: _renew_subs 2023-11-02T08:36:31.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e759e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e8142f20 con 0x7ff7e80b1870 2023-11-02T08:36:31.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e759e640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e759e640 10 monclient: _renew_subs 2023-11-02T08:36:31.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e759e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e8137d60 con 0x7ff7e80b1870 2023-11-02T08:36:31.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7ff7e759e640 1 librados: init done 2023-11-02T08:36:31.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7f795dad6640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6802/86123,v1:172.21.15.178:6803/86123] conn(0x7f793c0721f0 0x7f793c074610 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7f794dffb640 10 client.4272.objecter ms_handle_connect 0x7f793c0721f0 2023-11-02T08:36:31.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7f7a31d9b640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a2c0c51c0 0x7f7a2c02fbf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7a28000c00 tx=0x7f7a280047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.629+0000 7ff7e759e640 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:31.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.629+0000 7ff7e759e640 15 client.4321.objecter _session_command_op_assign -1 1 2023-11-02T08:36:31.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.629+0000 7ff7e759e640 20 client.4321.objecter _get_session osd=-1 returning homeless 2023-11-02T08:36:31.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.629+0000 7ff7e759e640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.629+0000 7ff7e759e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.629+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff7e8143470 con 0x7ff7e80b1870 2023-11-02T08:36:31.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.628+0000 7f7a22ffd640 10 client.?.objecter ms_handle_connect 0x7f7a2c0c51c0 2023-11-02T08:36:31.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.629+0000 7f7a22ffd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:31.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 1 -- 172.21.15.151:0/4225271198 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51000035f0 con 0x7f51040ac590 2023-11-02T08:36:31.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:31.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 10 monclient: dump: 2023-11-02T08:36:31.942 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.942 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.942 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.942 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.942 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.943 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.943 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.943 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.943 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.943 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 1 -- 172.21.15.151:0/4225271198 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5100003770 con 0x7f51040ac590 2023-11-02T08:36:31.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 1 -- 172.21.15.151:0/4225271198 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5100003b60 con 0x7f51040ac590 2023-11-02T08:36:31.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:31.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50f9ffb640 10 monclient: dump: 2023-11-02T08:36:31.945 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.946 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.946 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.946 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.946 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.946 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.947 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.947 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.947 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.947 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f50fb7fe640 4 set_mon_vals no callback set 2023-11-02T08:36:31.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f510a193640 5 monclient: authenticate success, global_id 4275 2023-11-02T08:36:31.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f510a193640 10 monclient: _renew_subs 2023-11-02T08:36:31.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f510a193640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:31.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51041502d0 con 0x7f51040ac590 2023-11-02T08:36:31.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f510a193640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f510a193640 10 monclient: _renew_subs 2023-11-02T08:36:31.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f510a193640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:31.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5104130a50 con 0x7f51040ac590 2023-11-02T08:36:31.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.638+0000 7f510a193640 1 librados: init done 2023-11-02T08:36:31.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.639+0000 7f510a193640 10 client.4275.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:31.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.639+0000 7f510a193640 15 client.4275.objecter _session_command_op_assign -1 1 2023-11-02T08:36:31.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.639+0000 7f510a193640 20 client.4275.objecter _get_session osd=-1 returning homeless 2023-11-02T08:36:31.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.639+0000 7f510a193640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.639+0000 7f510a193640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:31.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.639+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51040abaf0 con 0x7f51040ac590 2023-11-02T08:36:31.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 1 -- 172.21.15.151:0/3709476426 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28003ea0 con 0x7f7a2c0c51c0 2023-11-02T08:36:31.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:31.952 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 10 monclient: dump: 2023-11-02T08:36:31.952 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.952 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.952 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.952 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.953 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.953 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.953 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.953 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.953 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.954 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 1 -- 172.21.15.151:0/3709476426 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2803d620 con 0x7f7a2c0c51c0 2023-11-02T08:36:31.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 1 -- 172.21.15.151:0/3709476426 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2803d9b0 con 0x7f7a2c0c51c0 2023-11-02T08:36:31.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a3359e640 5 monclient: authenticate success, global_id 4324 2023-11-02T08:36:31.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a3359e640 10 monclient: _renew_subs 2023-11-02T08:36:31.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a3359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c0c6900 con 0x7f7a2c0c51c0 2023-11-02T08:36:31.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a30d99640 4 set_mon_vals no callback set 2023-11-02T08:36:31.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:31.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a22ffd640 10 monclient: dump: 2023-11-02T08:36:31.957 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.957 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.957 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.957 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.957 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.958 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.958 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.958 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.958 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.958 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a3359e640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a3359e640 10 monclient: _renew_subs 2023-11-02T08:36:31.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a3359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c0c6290 con 0x7f7a2c0c51c0 2023-11-02T08:36:31.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.644+0000 7f7a3359e640 1 librados: init done 2023-11-02T08:36:31.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.645+0000 7f7a3359e640 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:31.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.645+0000 7f7a3359e640 15 client.4324.objecter _session_command_op_assign -1 1 2023-11-02T08:36:31.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.645+0000 7f7a3359e640 20 client.4324.objecter _get_session osd=-1 returning homeless 2023-11-02T08:36:31.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.645+0000 7f7a3359e640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.645+0000 7f7a3359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:31.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.645+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7a2c02d3f0 con 0x7f7a2c0c51c0 2023-11-02T08:36:31.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.648+0000 7f9cf7fff640 1 -- 172.21.15.151:0/3515774077 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9d04044900 con 0x7f9d100a8b30 2023-11-02T08:36:31.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.649+0000 7f9cf7fff640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9cfc06bdc0 0x7f9cfc06e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.649+0000 7f9d0effd640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9cfc06bdc0 0x7f9cfc06e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.649+0000 7f9d0effd640 10 monclient: get_auth_request con 0x7f9cfc06bdc0 auth_method 0 2023-11-02T08:36:31.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.650+0000 7f9d0effd640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9cfc06bdc0 0x7f9cfc06e280 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009670 tx=0x7f9d00006bf0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.650+0000 7f9cf7fff640 10 client.4253.objecter ms_handle_connect 0x7f9cfc06bdc0 2023-11-02T08:36:31.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 1 -- 172.21.15.151:0/3515774077 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9d0402d040 con 0x7f9d100a8b30 2023-11-02T08:36:31.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 10 client.4253.objecter ms_dispatch 0x7f9d10000b90 osd_map(21..21 src has 1..21) v4 2023-11-02T08:36:31.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 3 client.4253.objecter handle_osd_map got epochs [21,21] > 0 2023-11-02T08:36:31.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 3 client.4253.objecter handle_osd_map decoding full epoch 21 2023-11-02T08:36:31.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 10 client.4253.objecter checking command 1 2023-11-02T08:36:31.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:6808/86098,v1:172.21.15.151:6809/86098] conn(0x7f9cfc0721a0 0x7f9cfc0745c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 20 client.4253.objecter _get_session s=0x7f9cfc071ad0 osd=0 3 2023-11-02T08:36:31.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 20 client.4253.objecter put_session s=0x7f9cfc071ad0 osd=0 3 2023-11-02T08:36:31.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 15 client.4253.objecter _session_command_op_remove -1 1 2023-11-02T08:36:31.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 20 client.4253.objecter _get_session s=0x7f9cfc071ad0 osd=0 3 2023-11-02T08:36:31.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 20 client.4253.objecter get_session s=0x7f9cfc071ad0 osd=0 3 2023-11-02T08:36:31.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 15 client.4253.objecter _session_command_op_assign 0 1 2023-11-02T08:36:31.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 20 client.4253.objecter put_session s=0x7f9cfc071ad0 osd=0 4 2023-11-02T08:36:31.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 10 client.4253.objecter _send_command 1 2023-11-02T08:36:31.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 1 -- 172.21.15.151:0/3515774077 --> [v2:172.21.15.151:6808/86098,v1:172.21.15.151:6809/86098] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9cfc074c70 con 0x7f9cfc0721a0 2023-11-02T08:36:31.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.652+0000 7f9cf7fff640 20 client.4253.objecter dump_active .. 0 homeless 2023-11-02T08:36:31.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.653+0000 7f9d0f7fe640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:6808/86098,v1:172.21.15.151:6809/86098] conn(0x7f9cfc0721a0 0x7f9cfc0745c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.653+0000 7f9d0f7fe640 10 monclient: get_auth_request con 0x7f9cfc0721a0 auth_method 0 2023-11-02T08:36:31.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.655+0000 7ff7d6ffd640 1 -- 172.21.15.151:0/4278818941 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff7e003eac0 con 0x7ff7e80b1870 2023-11-02T08:36:31.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.656+0000 7ff7d6ffd640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7ff7c806be10 0x7ff7c806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.656+0000 7ff7e659c640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7ff7c806be10 0x7ff7c806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.656+0000 7ff7e659c640 10 monclient: get_auth_request con 0x7ff7c806be10 auth_method 0 2023-11-02T08:36:31.969 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.656+0000 7ff7d6ffd640 10 client.4321.objecter ms_handle_connect 0x7ff7c806be10 2023-11-02T08:36:31.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.656+0000 7ff7e659c640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7ff7c806be10 0x7ff7c806e2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8003cc0 tx=0x7ff7d80079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 1 -- 172.21.15.151:0/4278818941 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff7e0050e70 con 0x7ff7e80b1870 2023-11-02T08:36:31.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 10 client.4321.objecter ms_dispatch 0x7ff7e8000b90 osd_map(21..21 src has 1..21) v4 2023-11-02T08:36:31.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 3 client.4321.objecter handle_osd_map got epochs [21,21] > 0 2023-11-02T08:36:31.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 3 client.4321.objecter handle_osd_map decoding full epoch 21 2023-11-02T08:36:31.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 10 client.4321.objecter checking command 1 2023-11-02T08:36:31.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.151:6824/86099,v1:172.21.15.151:6825/86099] conn(0x7ff7c80721f0 0x7ff7c8074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 20 client.4321.objecter _get_session s=0x7ff7c8071b20 osd=3 3 2023-11-02T08:36:31.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 20 client.4321.objecter put_session s=0x7ff7c8071b20 osd=3 3 2023-11-02T08:36:31.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 15 client.4321.objecter _session_command_op_remove -1 1 2023-11-02T08:36:31.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 20 client.4321.objecter _get_session s=0x7ff7c8071b20 osd=3 3 2023-11-02T08:36:31.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 20 client.4321.objecter get_session s=0x7ff7c8071b20 osd=3 3 2023-11-02T08:36:31.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 15 client.4321.objecter _session_command_op_assign 3 1 2023-11-02T08:36:31.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 20 client.4321.objecter put_session s=0x7ff7c8071b20 osd=3 4 2023-11-02T08:36:31.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 10 client.4321.objecter _send_command 1 2023-11-02T08:36:31.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 1 -- 172.21.15.151:0/4278818941 --> [v2:172.21.15.151:6824/86099,v1:172.21.15.151:6825/86099] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7c8074cc0 con 0x7ff7c80721f0 2023-11-02T08:36:31.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.657+0000 7ff7d6ffd640 20 client.4321.objecter dump_active .. 0 homeless 2023-11-02T08:36:31.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7ff7e5d9b640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.151:6824/86099,v1:172.21.15.151:6825/86099] conn(0x7ff7c80721f0 0x7ff7c8074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9cf7fff640 10 client.4253.objecter ms_handle_connect 0x7f9cfc0721a0 2023-11-02T08:36:31.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9d0f7fe640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:6808/86098,v1:172.21.15.151:6809/86098] conn(0x7f9cfc0721a0 0x7f9cfc0745c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 1 -- 172.21.15.151:0/2642604601 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f780035f0 con 0x7f9f80036e60 2023-11-02T08:36:31.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:31.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 10 monclient: dump: 2023-11-02T08:36:31.975 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.975 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.976 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.976 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.976 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.976 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.976 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.977 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.977 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.977 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 10 monclient: _finish_auth 0 2023-11-02T08:36:31.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 10 monclient: _check_auth_tickets 2023-11-02T08:36:31.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:31.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 1 -- 172.21.15.151:0/2642604601 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f78003770 con 0x7f9f80036e60 2023-11-02T08:36:31.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:31.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 1 -- 172.21.15.151:0/2642604601 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78003b60 con 0x7f9f80036e60 2023-11-02T08:36:31.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:31.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:31.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7dffb640 10 monclient: dump: 2023-11-02T08:36:31.979 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:31.979 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:31.979 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.980 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:31.980 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:31.980 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:31.980 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:31.980 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:31.981 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:31.981 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:31.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f8e3ac640 5 monclient: authenticate success, global_id 4259 2023-11-02T08:36:31.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f8e3ac640 10 monclient: _renew_subs 2023-11-02T08:36:31.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f8e3ac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8015d670 con 0x7f9f80036e60 2023-11-02T08:36:31.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f7f7fe640 4 set_mon_vals no callback set 2023-11-02T08:36:31.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f8e3ac640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f8e3ac640 10 monclient: _renew_subs 2023-11-02T08:36:31.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f8e3ac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.982 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f80034580 con 0x7f9f80036e60 2023-11-02T08:36:31.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7f9f8e3ac640 1 librados: init done 2023-11-02T08:36:31.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.658+0000 7ff7e5d9b640 10 monclient: get_auth_request con 0x7ff7c80721f0 auth_method 0 2023-11-02T08:36:31.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.659+0000 7f9f8e3ac640 10 client.4259.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:31.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.659+0000 7f9f8e3ac640 15 client.4259.objecter _session_command_op_assign -1 1 2023-11-02T08:36:31.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.660+0000 7f9f8e3ac640 20 client.4259.objecter _get_session osd=-1 returning homeless 2023-11-02T08:36:31.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.660+0000 7f9f8e3ac640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:31.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.660+0000 7f9f8e3ac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:31.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.660+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f8002d2d0 con 0x7f9f80036e60 2023-11-02T08:36:31.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.663+0000 7ff7d6ffd640 10 client.4321.objecter ms_handle_connect 0x7ff7c80721f0 2023-11-02T08:36:31.985 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.663+0000 7ff7e5d9b640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.151:6824/86099,v1:172.21.15.151:6825/86099] conn(0x7ff7c80721f0 0x7ff7c8074610 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.665+0000 7f898f7fe640 1 -- 172.21.15.151:0/3968162331 <== osd.2 v2:172.21.15.151:6816/86100 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8994074c70 con 0x7f89940721a0 2023-11-02T08:36:31.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.666+0000 7f898f7fe640 10 client.4318.objecter ms_dispatch 0x7f89a8000b90 command_reply(tid 1: 0 ) v1 2023-11-02T08:36:31.986 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.666+0000 7f898f7fe640 10 client.4318.objecter _finish_command 1 = system:0 2023-11-02T08:36:31.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.666+0000 7f898f7fe640 20 client.4318.objecter put_session s=0x7f8994071ad0 osd=2 4 2023-11-02T08:36:31.987 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.666+0000 7f898f7fe640 15 client.4318.objecter _session_command_op_remove 2 1 2023-11-02T08:36:31.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.687+0000 7f7cc27fc640 1 -- 172.21.15.151:0/141595971 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7ccc06b110 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.688+0000 7f7cc27fc640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7cb006be10 0x7f7cb006e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.688+0000 7f7cd1ba0640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7cb006be10 0x7f7cb006e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.988 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.688+0000 7f7cd1ba0640 10 monclient: get_auth_request con 0x7f7cb006be10 auth_method 0 2023-11-02T08:36:31.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.688+0000 7f7cc27fc640 10 client.4256.objecter ms_handle_connect 0x7f7cb006be10 2023-11-02T08:36:31.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.688+0000 7f7cd1ba0640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7cb006be10 0x7f7cb006e2d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc036000 tx=0x7f7cbc030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.989 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-02T08:36:31.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 15 client.4318.objecter _session_command_op_assign -1 2 2023-11-02T08:36:31.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 20 client.4318.objecter _get_session s=0x7f8994071ad0 osd=2 3 2023-11-02T08:36:31.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 20 client.4318.objecter put_session s=0x7f8994071ad0 osd=2 3 2023-11-02T08:36:31.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 20 client.4318.objecter _get_session s=0x7f8994071ad0 osd=2 3 2023-11-02T08:36:31.990 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 15 client.4318.objecter _session_command_op_remove -1 2 2023-11-02T08:36:31.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 20 client.4318.objecter get_session s=0x7f8994071ad0 osd=2 3 2023-11-02T08:36:31.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 15 client.4318.objecter _session_command_op_assign 2 2 2023-11-02T08:36:31.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 20 client.4318.objecter put_session s=0x7f8994071ad0 osd=2 4 2023-11-02T08:36:31.991 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 10 client.4318.objecter _send_command 2 2023-11-02T08:36:31.992 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.690+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 --> [v2:172.21.15.151:6816/86100,v1:172.21.15.151:6817/86100] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f89a8071720 con 0x7f89940721a0 2023-11-02T08:36:31.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 1 -- 172.21.15.151:0/141595971 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7ccc0d22f0 con 0x7f7cc40cc2e0 2023-11-02T08:36:31.993 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 10 client.4256.objecter ms_dispatch 0x7f7ccc000b90 osd_map(21..21 src has 1..21) v4 2023-11-02T08:36:31.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 3 client.4256.objecter handle_osd_map got epochs [21,21] > 0 2023-11-02T08:36:31.994 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 3 client.4256.objecter handle_osd_map decoding full epoch 21 2023-11-02T08:36:31.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 10 client.4256.objecter checking command 1 2023-11-02T08:36:31.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6818/86122,v1:172.21.15.178:6819/86122] conn(0x7f7cb00721f0 0x7f7cb0074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:31.995 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 20 client.4256.objecter _get_session s=0x7f7cb0071b20 osd=5 3 2023-11-02T08:36:31.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 20 client.4256.objecter put_session s=0x7f7cb0071b20 osd=5 3 2023-11-02T08:36:31.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 15 client.4256.objecter _session_command_op_remove -1 1 2023-11-02T08:36:31.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 20 client.4256.objecter _get_session s=0x7f7cb0071b20 osd=5 3 2023-11-02T08:36:31.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 20 client.4256.objecter get_session s=0x7f7cb0071b20 osd=5 3 2023-11-02T08:36:31.996 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 15 client.4256.objecter _session_command_op_assign 5 1 2023-11-02T08:36:31.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 20 client.4256.objecter put_session s=0x7f7cb0071b20 osd=5 4 2023-11-02T08:36:31.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 10 client.4256.objecter _send_command 1 2023-11-02T08:36:31.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 1 -- 172.21.15.151:0/141595971 --> [v2:172.21.15.178:6818/86122,v1:172.21.15.178:6819/86122] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7cb0074cc0 con 0x7f7cb00721f0 2023-11-02T08:36:31.997 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.693+0000 7f7cc27fc640 20 client.4256.objecter dump_active .. 0 homeless 2023-11-02T08:36:31.998 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.694+0000 7f7cd139f640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6818/86122,v1:172.21.15.178:6819/86122] conn(0x7f7cb00721f0 0x7f7cb0074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:31.998 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.694+0000 7f7cd139f640 10 monclient: get_auth_request con 0x7f7cb00721f0 auth_method 0 2023-11-02T08:36:31.998 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.699+0000 7f7cc27fc640 10 client.4256.objecter ms_handle_connect 0x7f7cb00721f0 2023-11-02T08:36:31.998 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.699+0000 7f7cd139f640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6818/86122,v1:172.21.15.178:6819/86122] conn(0x7f7cb00721f0 0x7f7cb0074610 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:31.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.700+0000 7f898f7fe640 1 -- 172.21.15.151:0/3968162331 <== osd.2 v2:172.21.15.151:6816/86100 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8998009280 con 0x7f89940721a0 2023-11-02T08:36:31.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.700+0000 7f898f7fe640 10 client.4318.objecter ms_dispatch 0x7f89a8000b90 command_reply(tid 2: 0 ) v1 2023-11-02T08:36:31.999 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.700+0000 7f898f7fe640 10 client.4318.objecter _finish_command 2 = system:0 2023-11-02T08:36:32.000 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.700+0000 7f898f7fe640 20 client.4318.objecter put_session s=0x7f8994071ad0 osd=2 4 2023-11-02T08:36:32.000 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.700+0000 7f898f7fe640 15 client.4318.objecter _session_command_op_remove 2 2 2023-11-02T08:36:32.002 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 10 librados: watch_flush enter 2023-11-02T08:36:32.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 10 librados: watch_flush exit 2023-11-02T08:36:32.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 10 client.4318.objecter close_session for osd.2 2023-11-02T08:36:32.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:6816/86100,v1:172.21.15.151:6817/86100] conn(0x7f89940721a0 msgr2=0x7f89940745c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.151:6816/86100,v1:172.21.15.151:6817/86100] conn(0x7f89940721a0 0x7f89940745c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 20 client.4318.objecter put_session s=0x7f8994071ad0 osd=2 1 2023-11-02T08:36:32.004 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 20 client.4318.objecter shutdown clearing up homeless session... 2023-11-02T08:36:32.005 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 10 client.4318.objecter successfully canceled tick 2023-11-02T08:36:32.005 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f899406bdc0 msgr2=0x7f899406e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f899406bdc0 0x7f899406e280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f89a000a400 tx=0x7f89a000d040 comp rx=0 tx=0).stop 2023-11-02T08:36:32.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 10 monclient: shutdown 2023-11-02T08:36:32.006 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 20 monclient: shutdown canceling and discarding version request 1 2023-11-02T08:36:32.007 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.007 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 msgr2=0x7f89a80abff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 --2- 172.21.15.151:0/3968162331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89a813ea80 0x7f89a80abff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f899c0090b0 tx=0x7f899c0046c0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89a759e640 1 -- 172.21.15.151:0/3968162331 reap_dead start 2023-11-02T08:36:32.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 shutdown_connections 2023-11-02T08:36:32.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 >> 172.21.15.151:0/3968162331 conn(0x7f89a8069e50 msgr2=0x7f89a813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 shutdown_connections 2023-11-02T08:36:32.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 -- 172.21.15.151:0/3968162331 wait complete. 2023-11-02T08:36:32.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.701+0000 7f89ad026640 1 librados: shutdown 2023-11-02T08:36:32.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.704+0000 7f9cf7fff640 1 -- 172.21.15.151:0/3515774077 <== osd.0 v2:172.21.15.151:6808/86098 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9cfc074c70 con 0x7f9cfc0721a0 2023-11-02T08:36:32.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.704+0000 7f9cf7fff640 10 client.4253.objecter ms_dispatch 0x7f9d10000b90 command_reply(tid 1: 0 ) v1 2023-11-02T08:36:32.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.704+0000 7f9cf7fff640 10 client.4253.objecter _finish_command 1 = system:0 2023-11-02T08:36:32.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.704+0000 7f9cf7fff640 20 client.4253.objecter put_session s=0x7f9cfc071ad0 osd=0 4 2023-11-02T08:36:32.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.704+0000 7f9cf7fff640 15 client.4253.objecter _session_command_op_remove 0 1 2023-11-02T08:36:32.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.708+0000 7f794dffb640 1 -- 172.21.15.151:0/2818850046 <== osd.7 v2:172.21.15.178:6802/86123 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f793c074cc0 con 0x7f793c0721f0 2023-11-02T08:36:32.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.708+0000 7f794dffb640 10 client.4272.objecter ms_dispatch 0x7f7958000b90 command_reply(tid 1: 0 ) v1 2023-11-02T08:36:32.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.708+0000 7f794dffb640 10 client.4272.objecter _finish_command 1 = system:0 2023-11-02T08:36:32.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.708+0000 7f794dffb640 20 client.4272.objecter put_session s=0x7f793c071b20 osd=7 4 2023-11-02T08:36:32.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.708+0000 7f794dffb640 15 client.4272.objecter _session_command_op_remove 7 1 2023-11-02T08:36:32.015 INFO:teuthology.orchestra.run.smithi151.stdout:60129542150 2023-11-02T08:36:32.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-02T08:36:32.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.719+0000 7ff7d6ffd640 1 -- 172.21.15.151:0/4278818941 <== osd.3 v2:172.21.15.151:6824/86099 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff7c8074cc0 con 0x7ff7c80721f0 2023-11-02T08:36:32.017 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.720+0000 7ff7d6ffd640 10 client.4321.objecter ms_dispatch 0x7ff7e8000b90 command_reply(tid 1: 0 ) v1 2023-11-02T08:36:32.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.720+0000 7ff7d6ffd640 10 client.4321.objecter _finish_command 1 = system:0 2023-11-02T08:36:32.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.720+0000 7ff7d6ffd640 20 client.4321.objecter put_session s=0x7ff7c8071b20 osd=3 4 2023-11-02T08:36:32.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.720+0000 7ff7d6ffd640 15 client.4321.objecter _session_command_op_remove 3 1 2023-11-02T08:36:32.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 10 client.4272.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-02T08:36:32.020 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 15 client.4272.objecter _session_command_op_assign -1 2 2023-11-02T08:36:32.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 20 client.4272.objecter _get_session s=0x7f793c071b20 osd=7 3 2023-11-02T08:36:32.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 20 client.4272.objecter put_session s=0x7f793c071b20 osd=7 3 2023-11-02T08:36:32.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 20 client.4272.objecter _get_session s=0x7f793c071b20 osd=7 3 2023-11-02T08:36:32.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 15 client.4272.objecter _session_command_op_remove -1 2 2023-11-02T08:36:32.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 20 client.4272.objecter get_session s=0x7f793c071b20 osd=7 3 2023-11-02T08:36:32.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 15 client.4272.objecter _session_command_op_assign 7 2 2023-11-02T08:36:32.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 20 client.4272.objecter put_session s=0x7f793c071b20 osd=7 4 2023-11-02T08:36:32.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 10 client.4272.objecter _send_command 2 2023-11-02T08:36:32.023 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.728+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 --> [v2:172.21.15.178:6802/86123,v1:172.21.15.178:6803/86123] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f79581451a0 con 0x7f793c0721f0 2023-11-02T08:36:32.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 10 client.4253.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-02T08:36:32.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 15 client.4253.objecter _session_command_op_assign -1 2 2023-11-02T08:36:32.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 20 client.4253.objecter _get_session s=0x7f9cfc071ad0 osd=0 3 2023-11-02T08:36:32.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 20 client.4253.objecter put_session s=0x7f9cfc071ad0 osd=0 3 2023-11-02T08:36:32.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 20 client.4253.objecter _get_session s=0x7f9cfc071ad0 osd=0 3 2023-11-02T08:36:32.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 15 client.4253.objecter _session_command_op_remove -1 2 2023-11-02T08:36:32.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 20 client.4253.objecter get_session s=0x7f9cfc071ad0 osd=0 3 2023-11-02T08:36:32.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 15 client.4253.objecter _session_command_op_assign 0 2 2023-11-02T08:36:32.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 20 client.4253.objecter put_session s=0x7f9cfc071ad0 osd=0 4 2023-11-02T08:36:32.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 10 client.4253.objecter _send_command 2 2023-11-02T08:36:32.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.732+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 --> [v2:172.21.15.151:6808/86098,v1:172.21.15.151:6809/86098] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9cf00011c0 con 0x7f9cfc0721a0 2023-11-02T08:36:32.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.739+0000 7f9cf7fff640 1 -- 172.21.15.151:0/3515774077 <== osd.0 v2:172.21.15.151:6808/86098 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9cf00011c0 con 0x7f9cfc0721a0 2023-11-02T08:36:32.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.739+0000 7f9cf7fff640 10 client.4253.objecter ms_dispatch 0x7f9d10000b90 command_reply(tid 2: 0 ) v1 2023-11-02T08:36:32.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.739+0000 7f9cf7fff640 10 client.4253.objecter _finish_command 2 = system:0 2023-11-02T08:36:32.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.739+0000 7f9cf7fff640 20 client.4253.objecter put_session s=0x7f9cfc071ad0 osd=0 4 2023-11-02T08:36:32.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.739+0000 7f9cf7fff640 15 client.4253.objecter _session_command_op_remove 0 2 2023-11-02T08:36:32.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 10 librados: watch_flush enter 2023-11-02T08:36:32.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 10 librados: watch_flush exit 2023-11-02T08:36:32.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 10 client.4253.objecter close_session for osd.0 2023-11-02T08:36:32.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:6808/86098,v1:172.21.15.151:6809/86098] conn(0x7f9cfc0721a0 msgr2=0x7f9cfc0745c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:6808/86098,v1:172.21.15.151:6809/86098] conn(0x7f9cfc0721a0 0x7f9cfc0745c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 20 client.4253.objecter put_session s=0x7f9cfc071ad0 osd=0 1 2023-11-02T08:36:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 20 client.4253.objecter shutdown clearing up homeless session... 2023-11-02T08:36:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 10 client.4253.objecter successfully canceled tick 2023-11-02T08:36:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9cfc06bdc0 msgr2=0x7f9cfc06e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9cfc06bdc0 0x7f9cfc06e280 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009670 tx=0x7f9d00006bf0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 10 monclient: shutdown 2023-11-02T08:36:32.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 20 monclient: shutdown canceling and discarding version request 1 2023-11-02T08:36:32.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 msgr2=0x7f9d1006f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 --2- 172.21.15.151:0/3515774077 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d100a8b30 0x7f9d1006f280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9d0402b950 tx=0x7f9d040046c0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d0f7fe640 1 -- 172.21.15.151:0/3515774077 reap_dead start 2023-11-02T08:36:32.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 shutdown_connections 2023-11-02T08:36:32.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 >> 172.21.15.151:0/3515774077 conn(0x7f9d100b2ad0 msgr2=0x7f9d100b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 shutdown_connections 2023-11-02T08:36:32.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 -- 172.21.15.151:0/3515774077 wait complete. 2023-11-02T08:36:32.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.740+0000 7f9d1596e640 1 librados: shutdown 2023-11-02T08:36:32.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.742+0000 7f794dffb640 1 -- 172.21.15.151:0/2818850046 <== osd.7 v2:172.21.15.178:6802/86123 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f79581451a0 con 0x7f793c0721f0 2023-11-02T08:36:32.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.742+0000 7f794dffb640 10 client.4272.objecter ms_dispatch 0x7f7958000b90 command_reply(tid 2: 0 ) v1 2023-11-02T08:36:32.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.742+0000 7f794dffb640 10 client.4272.objecter _finish_command 2 = system:0 2023-11-02T08:36:32.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.742+0000 7f794dffb640 20 client.4272.objecter put_session s=0x7f793c071b20 osd=7 4 2023-11-02T08:36:32.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.742+0000 7f794dffb640 15 client.4272.objecter _session_command_op_remove 7 2 2023-11-02T08:36:32.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 10 librados: watch_flush enter 2023-11-02T08:36:32.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 10 librados: watch_flush exit 2023-11-02T08:36:32.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 10 client.4272.objecter close_session for osd.7 2023-11-02T08:36:32.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6802/86123,v1:172.21.15.178:6803/86123] conn(0x7f793c0721f0 msgr2=0x7f793c074610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6802/86123,v1:172.21.15.178:6803/86123] conn(0x7f793c0721f0 0x7f793c074610 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 20 client.4272.objecter put_session s=0x7f793c071b20 osd=7 1 2023-11-02T08:36:32.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 20 client.4272.objecter shutdown clearing up homeless session... 2023-11-02T08:36:32.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 10 client.4272.objecter successfully canceled tick 2023-11-02T08:36:32.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f50f9ffb640 1 -- 172.21.15.151:0/4225271198 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5100003cc0 con 0x7f51040ac590 2023-11-02T08:36:32.037 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f793c06be10 msgr2=0x7f793c06e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f793c06be10 0x7f793c06e2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7954009670 tx=0x7f7954006bf0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 10 monclient: shutdown 2023-11-02T08:36:32.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79580b0950 msgr2=0x7f79581449b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 1 --2- 172.21.15.151:0/2818850046 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79580b0950 0x7f79581449b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f794800c430 tx=0x7f794800c8f0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795dad6640 1 -- 172.21.15.151:0/2818850046 reap_dead start 2023-11-02T08:36:32.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 shutdown_connections 2023-11-02T08:36:32.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.743+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 >> 172.21.15.151:0/2818850046 conn(0x7f7958065960 msgr2=0x7f7958065d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.744+0000 7f50f9ffb640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f50e806be10 0x7f50e806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.744+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 shutdown_connections 2023-11-02T08:36:32.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.744+0000 7f795f539640 1 -- 172.21.15.151:0/2818850046 wait complete. 2023-11-02T08:36:32.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.744+0000 7f795f539640 1 librados: shutdown 2023-11-02T08:36:32.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.744+0000 7f5109191640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f50e806be10 0x7f50e806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.744+0000 7f5109191640 10 monclient: get_auth_request con 0x7f50e806be10 auth_method 0 2023-11-02T08:36:32.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.744+0000 7f5109191640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f50e806be10 0x7f50e806e2d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50f4036000 tx=0x7f50f4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.744+0000 7f50f9ffb640 10 client.4275.objecter ms_handle_connect 0x7f50e806be10 2023-11-02T08:36:32.042 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-02T08:36:32.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 15 client.4321.objecter _session_command_op_assign -1 2 2023-11-02T08:36:32.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 20 client.4321.objecter _get_session s=0x7ff7c8071b20 osd=3 3 2023-11-02T08:36:32.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 20 client.4321.objecter put_session s=0x7ff7c8071b20 osd=3 3 2023-11-02T08:36:32.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 20 client.4321.objecter _get_session s=0x7ff7c8071b20 osd=3 3 2023-11-02T08:36:32.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 15 client.4321.objecter _session_command_op_remove -1 2 2023-11-02T08:36:32.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 20 client.4321.objecter get_session s=0x7ff7c8071b20 osd=3 3 2023-11-02T08:36:32.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 15 client.4321.objecter _session_command_op_assign 3 2 2023-11-02T08:36:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 20 client.4321.objecter put_session s=0x7ff7c8071b20 osd=3 4 2023-11-02T08:36:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 10 client.4321.objecter _send_command 2 2023-11-02T08:36:32.046 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.746+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 --> [v2:172.21.15.151:6824/86099,v1:172.21.15.151:6825/86099] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff7e806f790 con 0x7ff7c80721f0 2023-11-02T08:36:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.748+0000 7f7a22ffd640 1 -- 172.21.15.151:0/3709476426 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7a2803db10 con 0x7f7a2c0c51c0 2023-11-02T08:36:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.749+0000 7f7a22ffd640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7a14074620 0x7f7a14076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.750+0000 7f7a3259c640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7a14074620 0x7f7a14076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.750+0000 7f7a3259c640 10 monclient: get_auth_request con 0x7f7a14074620 auth_method 0 2023-11-02T08:36:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.750+0000 7f7a3259c640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7a14074620 0x7f7a14076ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7a24001050 tx=0x7f7a24001590 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.750+0000 7f7a22ffd640 10 client.4324.objecter ms_handle_connect 0x7f7a14074620 2023-11-02T08:36:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 1 -- 172.21.15.151:0/4225271198 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5100057970 con 0x7f51040ac590 2023-11-02T08:36:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 10 client.4275.objecter ms_dispatch 0x7f5104000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 3 client.4275.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:32.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 3 client.4275.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:32.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 10 client.4275.objecter checking command 1 2023-11-02T08:36:32.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x7f50e8072160 0x7f50e8074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 20 client.4275.objecter _get_session s=0x7f50e8071a90 osd=4 3 2023-11-02T08:36:32.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 20 client.4275.objecter put_session s=0x7f50e8071a90 osd=4 3 2023-11-02T08:36:32.050 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 15 client.4275.objecter _session_command_op_remove -1 1 2023-11-02T08:36:32.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 20 client.4275.objecter _get_session s=0x7f50e8071a90 osd=4 3 2023-11-02T08:36:32.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 20 client.4275.objecter get_session s=0x7f50e8071a90 osd=4 3 2023-11-02T08:36:32.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 15 client.4275.objecter _session_command_op_assign 4 1 2023-11-02T08:36:32.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 20 client.4275.objecter put_session s=0x7f50e8071a90 osd=4 4 2023-11-02T08:36:32.051 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 10 client.4275.objecter _send_command 1 2023-11-02T08:36:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 1 -- 172.21.15.151:0/4225271198 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50e8074c30 con 0x7f50e8072160 2023-11-02T08:36:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 20 client.4275.objecter dump_active .. 0 homeless 2023-11-02T08:36:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 1 -- 172.21.15.151:0/4225271198 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5100053cb0 con 0x7f51040ac590 2023-11-02T08:36:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.751+0000 7f50f9ffb640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-02T08:36:32.052 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.752+0000 7f5109992640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x7f50e8072160 0x7f50e8074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.752+0000 7f5109992640 10 monclient: get_auth_request con 0x7f50e8072160 auth_method 0 2023-11-02T08:36:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 1 -- 172.21.15.151:0/3709476426 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7a28079d80 con 0x7f7a2c0c51c0 2023-11-02T08:36:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 10 client.4324.objecter ms_dispatch 0x7f7a34000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 3 client.4324.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 3 client.4324.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 10 client.4324.objecter checking command 1 2023-11-02T08:36:32.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7ff7d6ffd640 1 -- 172.21.15.151:0/4278818941 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7e00773e0 con 0x7ff7e80b1870 2023-11-02T08:36:32.055 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7ff7d6ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-02T08:36:32.056 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-11-02T08:36:32.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-02T08:36:32.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6826/86121,v1:172.21.15.178:6827/86121] conn(0x7f7a1407aa00 0x7f7a1407ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 20 client.4324.objecter _get_session s=0x7f7a1407a330 osd=6 3 2023-11-02T08:36:32.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 20 client.4324.objecter put_session s=0x7f7a1407a330 osd=6 3 2023-11-02T08:36:32.057 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 15 client.4324.objecter _session_command_op_remove -1 1 2023-11-02T08:36:32.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 20 client.4324.objecter _get_session s=0x7f7a1407a330 osd=6 3 2023-11-02T08:36:32.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 20 client.4324.objecter get_session s=0x7f7a1407a330 osd=6 3 2023-11-02T08:36:32.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 15 client.4324.objecter _session_command_op_assign 6 1 2023-11-02T08:36:32.058 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 20 client.4324.objecter put_session s=0x7f7a1407a330 osd=6 4 2023-11-02T08:36:32.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 10 client.4324.objecter _send_command 1 2023-11-02T08:36:32.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 1 -- 172.21.15.151:0/3709476426 --> [v2:172.21.15.178:6826/86121,v1:172.21.15.178:6827/86121] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a1407d4d0 con 0x7f7a1407aa00 2023-11-02T08:36:32.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7f7a22ffd640 20 client.4324.objecter dump_active .. 0 homeless 2023-11-02T08:36:32.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7ff7d6ffd640 1 -- 172.21.15.151:0/4278818941 <== osd.3 v2:172.21.15.151:6824/86099 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff7dc009280 con 0x7ff7c80721f0 2023-11-02T08:36:32.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7ff7d6ffd640 10 client.4321.objecter ms_dispatch 0x7ff7e8000b90 command_reply(tid 2: 0 ) v1 2023-11-02T08:36:32.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7ff7d6ffd640 10 client.4321.objecter _finish_command 2 = system:0 2023-11-02T08:36:32.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7ff7d6ffd640 20 client.4321.objecter put_session s=0x7ff7c8071b20 osd=3 4 2023-11-02T08:36:32.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.754+0000 7ff7d6ffd640 15 client.4321.objecter _session_command_op_remove 3 2 2023-11-02T08:36:32.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7f7a38bc9640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6826/86121,v1:172.21.15.178:6827/86121] conn(0x7f7a1407aa00 0x7f7a1407ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7f7a22ffd640 1 -- 172.21.15.151:0/3709476426 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a28079b50 con 0x7f7a2c0c51c0 2023-11-02T08:36:32.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7f7a22ffd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-02T08:36:32.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7f7a38bc9640 10 monclient: get_auth_request con 0x7f7a1407aa00 auth_method 0 2023-11-02T08:36:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 10 librados: watch_flush enter 2023-11-02T08:36:32.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 10 librados: watch_flush exit 2023-11-02T08:36:32.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 10 client.4321.objecter close_session for osd.3 2023-11-02T08:36:32.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 >> [v2:172.21.15.151:6824/86099,v1:172.21.15.151:6825/86099] conn(0x7ff7c80721f0 msgr2=0x7ff7c8074610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.151:6824/86099,v1:172.21.15.151:6825/86099] conn(0x7ff7c80721f0 0x7ff7c8074610 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 20 client.4321.objecter put_session s=0x7ff7c8071b20 osd=3 1 2023-11-02T08:36:32.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 20 client.4321.objecter shutdown clearing up homeless session... 2023-11-02T08:36:32.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 10 client.4321.objecter successfully canceled tick 2023-11-02T08:36:32.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7ff7c806be10 msgr2=0x7ff7c806e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7ff7c806be10 0x7ff7c806e2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8003cc0 tx=0x7ff7d80079b0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 10 monclient: shutdown 2023-11-02T08:36:32.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e80b1870 msgr2=0x7ff7e8137820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 --2- 172.21.15.151:0/4278818941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7e80b1870 0x7ff7e8137820 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0036040 tx=0x7ff7e0004a40 comp rx=0 tx=0).stop 2023-11-02T08:36:32.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e6d9d640 1 -- 172.21.15.151:0/4278818941 reap_dead start 2023-11-02T08:36:32.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 shutdown_connections 2023-11-02T08:36:32.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 >> 172.21.15.151:0/4278818941 conn(0x7ff7e80b3970 msgr2=0x7ff7e8143830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 shutdown_connections 2023-11-02T08:36:32.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 -- 172.21.15.151:0/4278818941 wait complete. 2023-11-02T08:36:32.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.755+0000 7ff7e759e640 1 librados: shutdown 2023-11-02T08:36:32.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.756+0000 7f50f9ffb640 10 client.4275.objecter ms_handle_connect 0x7f50e8072160 2023-11-02T08:36:32.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.756+0000 7f5109992640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x7f50e8072160 0x7f50e8074580 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.068 INFO:teuthology.orchestra.run.smithi151.stdout:60129542150 2023-11-02T08:36:32.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-02T08:36:32.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.759+0000 7f7a22ffd640 10 client.4324.objecter ms_handle_connect 0x7f7a1407aa00 2023-11-02T08:36:32.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.759+0000 7f7a38bc9640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6826/86121,v1:172.21.15.178:6827/86121] conn(0x7f7a1407aa00 0x7f7a1407ce20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.071 INFO:teuthology.orchestra.run.smithi151.stdout:60129542148 2023-11-02T08:36:32.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-02T08:36:32.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.790+0000 7f9f7dffb640 1 -- 172.21.15.151:0/2642604601 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9f78003cc0 con 0x7f9f80036e60 2023-11-02T08:36:32.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.790+0000 7f9f7dffb640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f6c06be10 0x7f9f6c06e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.790+0000 7f9f8cba9640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f6c06be10 0x7f9f6c06e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.073 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.790+0000 7f9f8cba9640 10 monclient: get_auth_request con 0x7f9f6c06be10 auth_method 0 2023-11-02T08:36:32.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.791+0000 7f9f8cba9640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f6c06be10 0x7f9f6c06e2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f8002fdb0 tx=0x7f9f84009110 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.791+0000 7f9f7dffb640 10 client.4259.objecter ms_handle_connect 0x7f9f6c06be10 2023-11-02T08:36:32.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 1 -- 172.21.15.151:0/2642604601 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9f78058020 con 0x7f9f80036e60 2023-11-02T08:36:32.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 10 client.4259.objecter ms_dispatch 0x7f9f88000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:32.074 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 3 client.4259.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:32.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 3 client.4259.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:32.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 10 client.4259.objecter checking command 1 2023-11-02T08:36:32.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:6800/86101,v1:172.21.15.151:6801/86101] conn(0x7f9f6c0721f0 0x7f9f6c074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 20 client.4259.objecter _get_session s=0x7f9f6c071b20 osd=1 3 2023-11-02T08:36:32.075 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 20 client.4259.objecter put_session s=0x7f9f6c071b20 osd=1 3 2023-11-02T08:36:32.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 15 client.4259.objecter _session_command_op_remove -1 1 2023-11-02T08:36:32.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 20 client.4259.objecter _get_session s=0x7f9f6c071b20 osd=1 3 2023-11-02T08:36:32.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 20 client.4259.objecter get_session s=0x7f9f6c071b20 osd=1 3 2023-11-02T08:36:32.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 15 client.4259.objecter _session_command_op_assign 1 1 2023-11-02T08:36:32.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 20 client.4259.objecter put_session s=0x7f9f6c071b20 osd=1 4 2023-11-02T08:36:32.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 10 client.4259.objecter _send_command 1 2023-11-02T08:36:32.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 1 -- 172.21.15.151:0/2642604601 --> [v2:172.21.15.151:6800/86101,v1:172.21.15.151:6801/86101] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f6c074cc0 con 0x7f9f6c0721f0 2023-11-02T08:36:32.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f7dffb640 20 client.4259.objecter dump_active .. 0 homeless 2023-11-02T08:36:32.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.793+0000 7f9f8dbab640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:6800/86101,v1:172.21.15.151:6801/86101] conn(0x7f9f6c0721f0 0x7f9f6c074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.077 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.794+0000 7f9f8dbab640 10 monclient: get_auth_request con 0x7f9f6c0721f0 auth_method 0 2023-11-02T08:36:32.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.795+0000 7f7cc27fc640 1 -- 172.21.15.151:0/141595971 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ccc067340 con 0x7f7cc40cc2e0 2023-11-02T08:36:32.078 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.795+0000 7f7cc27fc640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-02T08:36:32.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.796+0000 7f9f7dffb640 1 -- 172.21.15.151:0/2642604601 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f78053c60 con 0x7f9f80036e60 2023-11-02T08:36:32.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.796+0000 7f9f7dffb640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-02T08:36:32.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.798+0000 7f9f7dffb640 10 client.4259.objecter ms_handle_connect 0x7f9f6c0721f0 2023-11-02T08:36:32.079 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.798+0000 7f9f8dbab640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:6800/86101,v1:172.21.15.151:6801/86101] conn(0x7f9f6c0721f0 0x7f9f6c074610 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.080 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.801+0000 7f50f9ffb640 1 -- 172.21.15.151:0/4225271198 <== osd.4 v2:172.21.15.178:6810/86120 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f50e8074c30 con 0x7f50e8072160 2023-11-02T08:36:32.080 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.801+0000 7f50f9ffb640 10 client.4275.objecter ms_dispatch 0x7f5104000b90 command_reply(tid 1: 0 ) v1 2023-11-02T08:36:32.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.801+0000 7f50f9ffb640 10 client.4275.objecter _finish_command 1 = system:0 2023-11-02T08:36:32.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.801+0000 7f50f9ffb640 20 client.4275.objecter put_session s=0x7f50e8071a90 osd=4 4 2023-11-02T08:36:32.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.801+0000 7f50f9ffb640 15 client.4275.objecter _session_command_op_remove 4 1 2023-11-02T08:36:32.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 10 client.4275.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-02T08:36:32.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 15 client.4275.objecter _session_command_op_assign -1 2 2023-11-02T08:36:32.082 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 20 client.4275.objecter _get_session s=0x7f50e8071a90 osd=4 3 2023-11-02T08:36:32.082 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 20 client.4275.objecter put_session s=0x7f50e8071a90 osd=4 3 2023-11-02T08:36:32.082 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 20 client.4275.objecter _get_session s=0x7f50e8071a90 osd=4 3 2023-11-02T08:36:32.082 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 15 client.4275.objecter _session_command_op_remove -1 2 2023-11-02T08:36:32.082 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 20 client.4275.objecter get_session s=0x7f50e8071a90 osd=4 3 2023-11-02T08:36:32.083 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 15 client.4275.objecter _session_command_op_assign 4 2 2023-11-02T08:36:32.083 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 20 client.4275.objecter put_session s=0x7f50e8071a90 osd=4 4 2023-11-02T08:36:32.083 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 10 client.4275.objecter _send_command 2 2023-11-02T08:36:32.083 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.815+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f51040b0440 con 0x7f50e8072160 2023-11-02T08:36:32.083 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.822+0000 7f50f9ffb640 1 -- 172.21.15.151:0/4225271198 <== osd.4 v2:172.21.15.178:6810/86120 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f50fc00f040 con 0x7f50e8072160 2023-11-02T08:36:32.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f50f9ffb640 10 client.4275.objecter ms_dispatch 0x7f5104000b90 command_reply(tid 2: 0 ) v1 2023-11-02T08:36:32.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f50f9ffb640 10 client.4275.objecter _finish_command 2 = system:0 2023-11-02T08:36:32.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f50f9ffb640 20 client.4275.objecter put_session s=0x7f50e8071a90 osd=4 4 2023-11-02T08:36:32.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f50f9ffb640 15 client.4275.objecter _session_command_op_remove 4 2 2023-11-02T08:36:32.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 10 librados: watch_flush enter 2023-11-02T08:36:32.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 10 librados: watch_flush exit 2023-11-02T08:36:32.086 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 10 client.4275.objecter close_session for osd.4 2023-11-02T08:36:32.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x7f50e8072160 msgr2=0x7f50e8074580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x7f50e8072160 0x7f50e8074580 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 20 client.4275.objecter put_session s=0x7f50e8071a90 osd=4 1 2023-11-02T08:36:32.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 20 client.4275.objecter shutdown clearing up homeless session... 2023-11-02T08:36:32.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 10 client.4275.objecter successfully canceled tick 2023-11-02T08:36:32.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f50e806be10 msgr2=0x7f50e806e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f50e806be10 0x7f50e806e2d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50f4036000 tx=0x7f50f4030000 comp rx=0 tx=0).stop 2023-11-02T08:36:32.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 10 monclient: shutdown 2023-11-02T08:36:32.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51040ac590 msgr2=0x7f5104138e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 --2- 172.21.15.151:0/4225271198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f51040ac590 0x7f5104138e40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f510000c410 tx=0x7f510000c810 comp rx=0 tx=0).stop 2023-11-02T08:36:32.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f5109992640 1 -- 172.21.15.151:0/4225271198 reap_dead start 2023-11-02T08:36:32.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 shutdown_connections 2023-11-02T08:36:32.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 >> 172.21.15.151:0/4225271198 conn(0x7f51040af560 msgr2=0x7f5104070900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 shutdown_connections 2023-11-02T08:36:32.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 -- 172.21.15.151:0/4225271198 wait complete. 2023-11-02T08:36:32.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.823+0000 7f510a193640 1 librados: shutdown 2023-11-02T08:36:32.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.826+0000 7f7a22ffd640 1 -- 172.21.15.151:0/3709476426 <== osd.6 v2:172.21.15.178:6826/86121 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f7a1407d4d0 con 0x7f7a1407aa00 2023-11-02T08:36:32.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.826+0000 7f7a22ffd640 10 client.4324.objecter ms_dispatch 0x7f7a34000b90 command_reply(tid 1: 0 ) v1 2023-11-02T08:36:32.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.826+0000 7f7a22ffd640 10 client.4324.objecter _finish_command 1 = system:0 2023-11-02T08:36:32.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.826+0000 7f7a22ffd640 20 client.4324.objecter put_session s=0x7f7a1407a330 osd=6 4 2023-11-02T08:36:32.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.826+0000 7f7a22ffd640 15 client.4324.objecter _session_command_op_remove 6 1 2023-11-02T08:36:32.092 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:32.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-02T08:36:32.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-02T08:36:32.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 15 client.4324.objecter _session_command_op_assign -1 2 2023-11-02T08:36:32.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 20 client.4324.objecter _get_session s=0x7f7a1407a330 osd=6 3 2023-11-02T08:36:32.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 20 client.4324.objecter put_session s=0x7f7a1407a330 osd=6 3 2023-11-02T08:36:32.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 20 client.4324.objecter _get_session s=0x7f7a1407a330 osd=6 3 2023-11-02T08:36:32.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 15 client.4324.objecter _session_command_op_remove -1 2 2023-11-02T08:36:32.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 20 client.4324.objecter get_session s=0x7f7a1407a330 osd=6 3 2023-11-02T08:36:32.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 15 client.4324.objecter _session_command_op_assign 6 2 2023-11-02T08:36:32.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 20 client.4324.objecter put_session s=0x7f7a1407a330 osd=6 4 2023-11-02T08:36:32.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 10 client.4324.objecter _send_command 2 2023-11-02T08:36:32.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.842+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 --> [v2:172.21.15.178:6826/86121,v1:172.21.15.178:6827/86121] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7a2c005960 con 0x7f7a1407aa00 2023-11-02T08:36:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.850+0000 7f9f7dffb640 1 -- 172.21.15.151:0/2642604601 <== osd.1 v2:172.21.15.151:6800/86101 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9f6c074cc0 con 0x7f9f6c0721f0 2023-11-02T08:36:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.850+0000 7f9f7dffb640 10 client.4259.objecter ms_dispatch 0x7f9f88000b90 command_reply(tid 1: 0 ) v1 2023-11-02T08:36:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.850+0000 7f9f7dffb640 10 client.4259.objecter _finish_command 1 = system:0 2023-11-02T08:36:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.851+0000 7f9f7dffb640 20 client.4259.objecter put_session s=0x7f9f6c071b20 osd=1 4 2023-11-02T08:36:32.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.851+0000 7f9f7dffb640 15 client.4259.objecter _session_command_op_remove 1 1 2023-11-02T08:36:32.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a22ffd640 1 -- 172.21.15.151:0/3709476426 <== osd.6 v2:172.21.15.178:6826/86121 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7a2c005960 con 0x7f7a1407aa00 2023-11-02T08:36:32.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a22ffd640 10 client.4324.objecter ms_dispatch 0x7f7a34000b90 command_reply(tid 2: 0 ) v1 2023-11-02T08:36:32.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a22ffd640 10 client.4324.objecter _finish_command 2 = system:0 2023-11-02T08:36:32.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a22ffd640 20 client.4324.objecter put_session s=0x7f7a1407a330 osd=6 4 2023-11-02T08:36:32.098 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a22ffd640 15 client.4324.objecter _session_command_op_remove 6 2 2023-11-02T08:36:32.099 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 10 librados: watch_flush enter 2023-11-02T08:36:32.099 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 10 librados: watch_flush exit 2023-11-02T08:36:32.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 10 client.4324.objecter close_session for osd.6 2023-11-02T08:36:32.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6826/86121,v1:172.21.15.178:6827/86121] conn(0x7f7a1407aa00 msgr2=0x7f7a1407ce20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6826/86121,v1:172.21.15.178:6827/86121] conn(0x7f7a1407aa00 0x7f7a1407ce20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 20 client.4324.objecter put_session s=0x7f7a1407a330 osd=6 1 2023-11-02T08:36:32.100 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 20 client.4324.objecter shutdown clearing up homeless session... 2023-11-02T08:36:32.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 10 client.4324.objecter successfully canceled tick 2023-11-02T08:36:32.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7a14074620 msgr2=0x7f7a14076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7a14074620 0x7f7a14076ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7a24001050 tx=0x7f7a24001590 comp rx=0 tx=0).stop 2023-11-02T08:36:32.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 10 monclient: shutdown 2023-11-02T08:36:32.101 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a2c0c51c0 msgr2=0x7f7a2c02fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 1 --2- 172.21.15.151:0/3709476426 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a2c0c51c0 0x7f7a2c02fbf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7a28000c00 tx=0x7f7a280047d0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a38bc9640 1 -- 172.21.15.151:0/3709476426 reap_dead start 2023-11-02T08:36:32.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 shutdown_connections 2023-11-02T08:36:32.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.854+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 >> 172.21.15.151:0/3709476426 conn(0x7f7a2c006330 msgr2=0x7f7a2c0beb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.855+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 shutdown_connections 2023-11-02T08:36:32.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.855+0000 7f7a3359e640 1 -- 172.21.15.151:0/3709476426 wait complete. 2023-11-02T08:36:32.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.855+0000 7f7a3359e640 1 librados: shutdown 2023-11-02T08:36:32.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 10 client.4259.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-02T08:36:32.104 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 15 client.4259.objecter _session_command_op_assign -1 2 2023-11-02T08:36:32.104 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 20 client.4259.objecter _get_session s=0x7f9f6c071b20 osd=1 3 2023-11-02T08:36:32.104 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 20 client.4259.objecter put_session s=0x7f9f6c071b20 osd=1 3 2023-11-02T08:36:32.104 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 20 client.4259.objecter _get_session s=0x7f9f6c071b20 osd=1 3 2023-11-02T08:36:32.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 15 client.4259.objecter _session_command_op_remove -1 2 2023-11-02T08:36:32.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 20 client.4259.objecter get_session s=0x7f9f6c071b20 osd=1 3 2023-11-02T08:36:32.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 15 client.4259.objecter _session_command_op_assign 1 2 2023-11-02T08:36:32.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 20 client.4259.objecter put_session s=0x7f9f6c071b20 osd=1 4 2023-11-02T08:36:32.105 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 10 client.4259.objecter _send_command 2 2023-11-02T08:36:32.106 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:32.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-02T08:36:32.107 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.867+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 --> [v2:172.21.15.151:6800/86101,v1:172.21.15.151:6801/86101] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9f8000a2e0 con 0x7f9f6c0721f0 2023-11-02T08:36:32.108 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f7dffb640 1 -- 172.21.15.151:0/2642604601 <== osd.1 v2:172.21.15.151:6800/86101 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9f8000a2e0 con 0x7f9f6c0721f0 2023-11-02T08:36:32.108 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f7dffb640 10 client.4259.objecter ms_dispatch 0x7f9f88000b90 command_reply(tid 2: 0 ) v1 2023-11-02T08:36:32.108 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f7dffb640 10 client.4259.objecter _finish_command 2 = system:0 2023-11-02T08:36:32.108 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f7dffb640 20 client.4259.objecter put_session s=0x7f9f6c071b20 osd=1 4 2023-11-02T08:36:32.109 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f7dffb640 15 client.4259.objecter _session_command_op_remove 1 2 2023-11-02T08:36:32.111 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f8e3ac640 10 librados: watch_flush enter 2023-11-02T08:36:32.111 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f8e3ac640 10 librados: watch_flush exit 2023-11-02T08:36:32.111 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f8e3ac640 10 client.4259.objecter close_session for osd.1 2023-11-02T08:36:32.111 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:6800/86101,v1:172.21.15.151:6801/86101] conn(0x7f9f6c0721f0 msgr2=0x7f9f6c074610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.111 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f8e3ac640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:6800/86101,v1:172.21.15.151:6801/86101] conn(0x7f9f6c0721f0 0x7f9f6c074610 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.112 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f8e3ac640 20 client.4259.objecter put_session s=0x7f9f6c071b20 osd=1 1 2023-11-02T08:36:32.112 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f8e3ac640 20 client.4259.objecter shutdown clearing up homeless session... 2023-11-02T08:36:32.112 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.884+0000 7f9f8e3ac640 10 client.4259.objecter successfully canceled tick 2023-11-02T08:36:32.112 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f6c06be10 msgr2=0x7f9f6c06e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.112 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f9f6c06be10 0x7f9f6c06e2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f8002fdb0 tx=0x7f9f84009110 comp rx=0 tx=0).stop 2023-11-02T08:36:32.113 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 10 monclient: shutdown 2023-11-02T08:36:32.113 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.113 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f80036e60 msgr2=0x7f9f800c8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.113 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 1 --2- 172.21.15.151:0/2642604601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9f80036e60 0x7f9f800c8350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800c410 tx=0x7f9f7800c810 comp rx=0 tx=0).stop 2023-11-02T08:36:32.113 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8dbab640 1 -- 172.21.15.151:0/2642604601 reap_dead start 2023-11-02T08:36:32.114 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 shutdown_connections 2023-11-02T08:36:32.114 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 >> 172.21.15.151:0/2642604601 conn(0x7f9f800062b0 msgr2=0x7f9f800c06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.114 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 shutdown_connections 2023-11-02T08:36:32.114 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 1 -- 172.21.15.151:0/2642604601 wait complete. 2023-11-02T08:36:32.114 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.885+0000 7f9f8e3ac640 1 librados: shutdown 2023-11-02T08:36:32.115 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:32.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-02T08:36:32.116 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.898+0000 7f7cc27fc640 1 -- 172.21.15.151:0/141595971 <== osd.5 v2:172.21.15.178:6818/86122 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f7cb0074cc0 con 0x7f7cb00721f0 2023-11-02T08:36:32.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.898+0000 7f7cc27fc640 10 client.4256.objecter ms_dispatch 0x7f7ccc000b90 command_reply(tid 1: 0 ) v1 2023-11-02T08:36:32.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.898+0000 7f7cc27fc640 10 client.4256.objecter _finish_command 1 = system:0 2023-11-02T08:36:32.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.898+0000 7f7cc27fc640 20 client.4256.objecter put_session s=0x7f7cb0071b20 osd=5 4 2023-11-02T08:36:32.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.898+0000 7f7cc27fc640 15 client.4256.objecter _session_command_op_remove 5 1 2023-11-02T08:36:32.117 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 10 client.4256.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-11-02T08:36:32.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 15 client.4256.objecter _session_command_op_assign -1 2 2023-11-02T08:36:32.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 20 client.4256.objecter _get_session s=0x7f7cb0071b20 osd=5 3 2023-11-02T08:36:32.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 20 client.4256.objecter put_session s=0x7f7cb0071b20 osd=5 3 2023-11-02T08:36:32.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 20 client.4256.objecter _get_session s=0x7f7cb0071b20 osd=5 3 2023-11-02T08:36:32.118 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 15 client.4256.objecter _session_command_op_remove -1 2 2023-11-02T08:36:32.119 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 20 client.4256.objecter get_session s=0x7f7cb0071b20 osd=5 3 2023-11-02T08:36:32.119 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 15 client.4256.objecter _session_command_op_assign 5 2 2023-11-02T08:36:32.119 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 20 client.4256.objecter put_session s=0x7f7cb0071b20 osd=5 4 2023-11-02T08:36:32.119 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 10 client.4256.objecter _send_command 2 2023-11-02T08:36:32.119 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.911+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 --> [v2:172.21.15.178:6818/86122,v1:172.21.15.178:6819/86122] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7cc402c0b0 con 0x7f7cb00721f0 2023-11-02T08:36:32.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cc27fc640 1 -- 172.21.15.151:0/141595971 <== osd.5 v2:172.21.15.178:6818/86122 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7cc402c0b0 con 0x7f7cb00721f0 2023-11-02T08:36:32.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cc27fc640 10 client.4256.objecter ms_dispatch 0x7f7ccc000b90 command_reply(tid 2: 0 ) v1 2023-11-02T08:36:32.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cc27fc640 10 client.4256.objecter _finish_command 2 = system:0 2023-11-02T08:36:32.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cc27fc640 20 client.4256.objecter put_session s=0x7f7cb0071b20 osd=5 4 2023-11-02T08:36:32.120 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cc27fc640 15 client.4256.objecter _session_command_op_remove 5 2 2023-11-02T08:36:32.123 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cd2ba2640 10 librados: watch_flush enter 2023-11-02T08:36:32.123 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cd2ba2640 10 librados: watch_flush exit 2023-11-02T08:36:32.123 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cd2ba2640 10 client.4256.objecter close_session for osd.5 2023-11-02T08:36:32.123 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6818/86122,v1:172.21.15.178:6819/86122] conn(0x7f7cb00721f0 msgr2=0x7f7cb0074610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cd2ba2640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6818/86122,v1:172.21.15.178:6819/86122] conn(0x7f7cb00721f0 0x7f7cb0074610 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cd2ba2640 20 client.4256.objecter put_session s=0x7f7cb0071b20 osd=5 1 2023-11-02T08:36:32.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cd2ba2640 20 client.4256.objecter shutdown clearing up homeless session... 2023-11-02T08:36:32.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.949+0000 7f7cd2ba2640 10 client.4256.objecter successfully canceled tick 2023-11-02T08:36:32.124 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7cb006be10 msgr2=0x7f7cb006e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f7cb006be10 0x7f7cb006e2d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc036000 tx=0x7f7cbc030000 comp rx=0 tx=0).stop 2023-11-02T08:36:32.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 10 monclient: shutdown 2023-11-02T08:36:32.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc40cc2e0 msgr2=0x7f7cc4037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.125 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 1 --2- 172.21.15.151:0/141595971 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7cc40cc2e0 0x7f7cc4037fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc02c840 tx=0x7f7ccc067960 comp rx=0 tx=0).stop 2023-11-02T08:36:32.126 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd23a1640 1 -- 172.21.15.151:0/141595971 reap_dead start 2023-11-02T08:36:32.126 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 shutdown_connections 2023-11-02T08:36:32.126 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 >> 172.21.15.151:0/141595971 conn(0x7f7cc40061c0 msgr2=0x7f7cc40c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.126 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 shutdown_connections 2023-11-02T08:36:32.126 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 1 -- 172.21.15.151:0/141595971 wait complete. 2023-11-02T08:36:32.127 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:31.950+0000 7f7cd2ba2640 1 librados: shutdown 2023-11-02T08:36:32.127 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:32.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-02T08:36:32.308 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.305+0000 7f4ee15d8640 10 monclient: get_monmap_and_config 2023-11-02T08:36:32.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.305+0000 7f4ee15d8640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.309 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.305+0000 7f4ee15d8640 10 monclient: monmap: 2023-11-02T08:36:32.309 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.309 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.310 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:32.306559+0000 2023-11-02T08:36:32.310 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:32.306559+0000 2023-11-02T08:36:32.310 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.310 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.311 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:32.311 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:32.311 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:32.311 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:32.311 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:32.312 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.306+0000 7f4ee15d8640 1 Processor -- start 2023-11-02T08:36:32.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.306+0000 7f4ee15d8640 1 -- start start 2023-11-02T08:36:32.313 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.306+0000 7f4ee15d8640 10 monclient: init 2023-11-02T08:36:32.314 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.314 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-11-02T08:36:32.314 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ed4038f10 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:32.314 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ed40392f0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:32.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4ed40357e0 0x7f4ed40c8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed40357e0 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:32.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed4034080 con 0x7f4ed4038f10 2023-11-02T08:36:32.315 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed4037370 con 0x7f4ed40392f0 2023-11-02T08:36:32.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed4037650 con 0x7f4ed40357e0 2023-11-02T08:36:32.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.316 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f4ee15d8640 10 monclient(hunting): authenticate will time out at 1363.663306s 2023-11-02T08:36:32.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f2f51841640 10 monclient: get_monmap_and_config 2023-11-02T08:36:32.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f2f51841640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.317 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.307+0000 7f2f51841640 10 monclient: monmap: 2023-11-02T08:36:32.318 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.318 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.318 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:32.308837+0000 2023-11-02T08:36:32.318 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:32.308837+0000 2023-11-02T08:36:32.318 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.319 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.319 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:32.319 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:32.319 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:32.320 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:32.320 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:32.320 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7f4edb7fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f4ed40392f0 0x7f4ed40c7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:42884/0 (socket says 172.21.15.151:42884) 2023-11-02T08:36:32.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7f4edb7fe640 1 -- 172.21.15.151:0/1971656940 learned_addr learned my addr 172.21.15.151:0/1971656940 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:32.321 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7f537359e640 10 monclient: get_monmap_and_config 2023-11-02T08:36:32.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7f537359e640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.322 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7f537359e640 10 monclient: monmap: 2023-11-02T08:36:32.322 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.322 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.322 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:32.309247+0000 2023-11-02T08:36:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:32.309247+0000 2023-11-02T08:36:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:32.323 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:32.324 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:32.324 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:32.324 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:32.324 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7f4ee0dd7640 1 --2- 172.21.15.151:0/1971656940 >> v2:172.21.15.151:3300/0 conn(0x7f4ed40357e0 0x7f4ed40c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7faffab7d640 10 monclient: get_monmap_and_config 2023-11-02T08:36:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7faffab7d640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7faffab7d640 10 monclient: monmap: 2023-11-02T08:36:32.326 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.326 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.326 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:32.310039+0000 2023-11-02T08:36:32.326 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:32.310039+0000 2023-11-02T08:36:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:32.327 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:32.328 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:32.328 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7fb731ff6640 10 monclient: get_monmap_and_config 2023-11-02T08:36:32.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.308+0000 7fb731ff6640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.309+0000 7fb731ff6640 10 monclient: monmap: 2023-11-02T08:36:32.329 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.329 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.329 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:32.310101+0000 2023-11-02T08:36:32.330 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:32.310101+0000 2023-11-02T08:36:32.330 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.330 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.330 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:32.330 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:32.331 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:32.331 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:32.331 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:32.331 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.309+0000 7f2f51841640 1 Processor -- start 2023-11-02T08:36:32.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.309+0000 7f2f51841640 1 -- start start 2023-11-02T08:36:32.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.309+0000 7f2f51841640 10 monclient: init 2023-11-02T08:36:32.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f4ee0dd7640 10 monclient(hunting): get_auth_request con 0x7f4ed40357e0 auth_method 0 2023-11-02T08:36:32.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f4ee0dd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f4ee0dd7640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f4ee0dd7640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.333 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.309+0000 7f537359e640 1 Processor -- start 2023-11-02T08:36:32.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.309+0000 7f537359e640 1 -- start start 2023-11-02T08:36:32.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.309+0000 7f537359e640 10 monclient: init 2023-11-02T08:36:32.334 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f2f51841640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-11-02T08:36:32.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f2f51841640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-11-02T08:36:32.336 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53740a8720 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:32.336 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f2f51841640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c0b75f0 0x7f2f4c0a8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.336 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7fb731ff6640 1 Processor -- start 2023-11-02T08:36:32.337 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f537413e3b0 0x7f53740ad840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.337 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 10 monclient(hunting): picked mon.noname-c con 0x7f537413e3b0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:32.337 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f2f51841640 10 monclient(hunting): picked mon.noname-b con 0x7f2f4c0b75f0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.338 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f2f51841640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2f4c0a8c90 0x7f2f4c13e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.339 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f53740add80 0x7f53740ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.339 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 10 monclient(hunting): picked mon.noname-a con 0x7f53740add80 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:32.339 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53740a9fc0 con 0x7f53740a8720 2023-11-02T08:36:32.339 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f537413e820 con 0x7f53740add80 2023-11-02T08:36:32.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53740a8b00 con 0x7f537413e3b0 2023-11-02T08:36:32.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f537359e640 10 monclient(hunting): authenticate will time out at 1363.666794s 2023-11-02T08:36:32.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f2f51841640 10 monclient(hunting): picked mon.noname-a con 0x7f2f4c0a8c90 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:32.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7f5371d9b640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f537413e3b0 0x7f53740ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.341 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f2f51841640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f4c13ec50 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:32.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f5371d9b640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f537413e3b0 0x7f53740ad840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36268/0 (socket says 172.21.15.151:36268) 2023-11-02T08:36:32.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f2f51841640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c0aa600 con 0x7f2f4c13ec50 2023-11-02T08:36:32.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f2f51841640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f2f51841640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4c0a9660 con 0x7f2f4c0a8c90 2023-11-02T08:36:32.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f2f51841640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f2f51841640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c0b79d0 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f2f51841640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f2f51841640 10 monclient(hunting): authenticate will time out at 1363.667271s 2023-11-02T08:36:32.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7fb731ff6640 1 -- start start 2023-11-02T08:36:32.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.310+0000 7fb731ff6640 10 monclient: init 2023-11-02T08:36:32.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-11-02T08:36:32.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb7240c53e0 0x7fb724038f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 10 monclient(hunting): picked mon.noname-c con 0x7fb7240c53e0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:32.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb724036360 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:32.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb7240cbb50 0x7fb7240d2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 10 monclient(hunting): picked mon.noname-a con 0x7fb7240cbb50 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:32.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb724036040 con 0x7fb724036360 2023-11-02T08:36:32.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb724037670 con 0x7fb7240cbb50 2023-11-02T08:36:32.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb724035cc0 con 0x7fb7240c53e0 2023-11-02T08:36:32.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.348 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7fb731ff6640 10 monclient(hunting): authenticate will time out at 1363.667274s 2023-11-02T08:36:32.348 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f4edaffd640 1 -- 172.21.15.151:0/1971656940 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655612294 0 0) 0x7f4ed4037370 con 0x7f4ed40392f0 2023-11-02T08:36:32.348 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f4edaffd640 10 monclient(hunting): _negotiate 2023-11-02T08:36:32.349 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f4edaffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.349 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f4edaffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.349 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f4edaffd640 10 monclient(hunting): my global_id is 4327 2023-11-02T08:36:32.350 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f4edaffd640 1 -- 172.21.15.151:0/1971656940 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ecc003520 con 0x7f4ed40392f0 2023-11-02T08:36:32.350 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f2f4affd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c0b75f0 0x7f2f4c0a8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.351 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f5371d9b640 1 -- 172.21.15.151:0/2004493105 learned_addr learned my addr 172.21.15.151:0/2004493105 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.351 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f5371d9b640 10 monclient(hunting): get_auth_request con 0x7f537413e3b0 auth_method 0 2023-11-02T08:36:32.352 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7faffab7d640 1 Processor -- start 2023-11-02T08:36:32.352 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f5371d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.353 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f5371d9b640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.353 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.311+0000 7f5371d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.353 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f2f4affd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c0b75f0 0x7f2f4c0a8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54800/0 (socket says 172.21.15.151:54800) 2023-11-02T08:36:32.354 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f2f4affd640 1 -- 172.21.15.151:0/3041811067 learned_addr learned my addr 172.21.15.151:0/3041811067 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.354 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f2f4affd640 10 monclient(hunting): get_auth_request con 0x7f2f4c0b75f0 auth_method 0 2023-11-02T08:36:32.354 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f2f4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.354 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f2f4affd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.354 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f2f4affd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 1 -- start start 2023-11-02T08:36:32.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 10 monclient: init 2023-11-02T08:36:32.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f4ee0dd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f4ee0dd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f4ee0dd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f2f4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f2f4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f2f4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f6498e1b640 10 monclient: get_monmap_and_config 2023-11-02T08:36:32.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f6498e1b640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7f6498e1b640 10 monclient: monmap: 2023-11-02T08:36:32.358 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.358 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.358 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:32.313947+0000 2023-11-02T08:36:32.358 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:32.313947+0000 2023-11-02T08:36:32.358 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.359 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.359 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:32.359 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:32.359 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:32.359 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:32.359 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:32.360 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.360 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-11-02T08:36:32.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec006fd0 0x7fafec0355e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 10 monclient(hunting): picked mon.noname-b con 0x7fafec006fd0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafec0073b0 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:32.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafec030ae0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:32.361 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafec035d10 con 0x7fafec0073b0 2023-11-02T08:36:32.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafec037570 con 0x7fafec030ae0 2023-11-02T08:36:32.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0337a0 con 0x7fafec006fd0 2023-11-02T08:36:32.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.362 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.312+0000 7faffab7d640 10 monclient(hunting): authenticate will time out at 1363.668790s 2023-11-02T08:36:32.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7faff8919640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec006fd0 0x7fafec0355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7f6498e1b640 1 Processor -- start 2023-11-02T08:36:32.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7f6498e1b640 1 -- start start 2023-11-02T08:36:32.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7f6498e1b640 10 monclient: init 2023-11-02T08:36:32.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7faff8919640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec006fd0 0x7fafec0355e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54810/0 (socket says 172.21.15.151:54810) 2023-11-02T08:36:32.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7faff8919640 1 -- 172.21.15.151:0/2592036571 learned_addr learned my addr 172.21.15.151:0/2592036571 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7faff8919640 10 monclient(hunting): get_auth_request con 0x7fafec006fd0 auth_method 0 2023-11-02T08:36:32.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7faff8919640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7faff8919640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.313+0000 7faff8919640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.314+0000 7f6498e1b640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.314+0000 7f6498e1b640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-11-02T08:36:32.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.314+0000 7f6498e1b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f649406f810 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:32.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.314+0000 7f6498e1b640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f64940af750 0x7f649406fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.314+0000 7f6498e1b640 10 monclient(hunting): picked mon.noname-a con 0x7f64940af750 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:32.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f6498e1b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f649413c6e0 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:32.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f5371d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.367 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f5371d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f5371d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f6498e1b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649413f630 con 0x7f649413c6e0 2023-11-02T08:36:32.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f6498e1b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6494070bf0 con 0x7f649406f810 2023-11-02T08:36:32.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f6498e1b640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f6498e1b640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6494071780 con 0x7f64940af750 2023-11-02T08:36:32.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f6498e1b640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.315+0000 7f6498e1b640 10 monclient(hunting): authenticate will time out at 1363.671038s 2023-11-02T08:36:32.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f4ee0dd7640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2023-11-02T08:36:32.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f4ee0dd7640 1 -- 172.21.15.151:0/1971656940 >> v1:172.21.15.178:6789/0 conn(0x7f4ed40392f0 legacy=0x7f4ed40c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f4ee0dd7640 1 -- 172.21.15.151:0/1971656940 >> v1:172.21.15.151:6789/0 conn(0x7f4ed4038f10 legacy=0x7f4ed4033970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f4ee0dd7640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f4ee0dd7640 1 monclient: found mon.noname-a 2023-11-02T08:36:32.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f4ee0dd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f4ee0dd7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:32.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f4ee0dd7640 1 -- 172.21.15.151:0/1971656940 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed4035f40 con 0x7f4ed40357e0 2023-11-02T08:36:32.372 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f5371d9b640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2023-11-02T08:36:32.372 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f5371d9b640 1 -- 172.21.15.151:0/2004493105 >> v2:172.21.15.151:3300/0 conn(0x7f53740add80 msgr2=0x7f53740ae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.372 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f5371d9b640 1 --2- 172.21.15.151:0/2004493105 >> v2:172.21.15.151:3300/0 conn(0x7f53740add80 0x7f53740ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f5371d9b640 1 -- 172.21.15.151:0/2004493105 >> v1:172.21.15.151:6789/0 conn(0x7f53740a8720 legacy=0x7f537413dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f5371d9b640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f5371d9b640 1 monclient: found mon.noname-c 2023-11-02T08:36:32.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f5371d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f5371d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.316+0000 7f5371d9b640 1 -- 172.21.15.151:0/2004493105 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53740aaa00 con 0x7f537413e3b0 2023-11-02T08:36:32.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7f2f4affd640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2023-11-02T08:36:32.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7f2f4affd640 1 -- 172.21.15.151:0/3041811067 >> v2:172.21.15.151:3300/0 conn(0x7f2f4c0a8c90 msgr2=0x7f2f4c13e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7f2f4affd640 1 --2- 172.21.15.151:0/3041811067 >> v2:172.21.15.151:3300/0 conn(0x7f2f4c0a8c90 0x7f2f4c13e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7f2f4affd640 1 -- 172.21.15.151:0/3041811067 >> v1:172.21.15.178:6789/0 conn(0x7f2f4c13ec50 legacy=0x7f2f4c1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7f2f4affd640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7f2f4affd640 1 monclient: found mon.noname-b 2023-11-02T08:36:32.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7f2f4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7f2f4affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7f2f4affd640 1 -- 172.21.15.151:0/3041811067 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c0a9140 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7fb72b7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb7240c53e0 0x7fb724038f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7faff8919640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7faff8919640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.317+0000 7faff8919640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7f4ee0dd7640 1 --2- 172.21.15.151:0/1971656940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ed40357e0 0x7f4ed40c8500 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4edc068260 tx=0x7f4edc0a5580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896b7795b7446a37 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7fc4a49f0640 10 monclient: get_monmap_and_config 2023-11-02T08:36:32.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7fc4a49f0640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7fc4a49f0640 10 monclient: monmap: 2023-11-02T08:36:32.379 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.379 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.380 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:32.320431+0000 2023-11-02T08:36:32.380 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:32.320431+0000 2023-11-02T08:36:32.380 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.380 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.380 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:32.381 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:32.381 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:32.381 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:32.381 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:32.381 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7faff8919640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2023-11-02T08:36:32.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7faff8919640 1 -- 172.21.15.151:0/2592036571 >> v1:172.21.15.178:6789/0 conn(0x7fafec030ae0 legacy=0x7fafec0361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7faff8919640 1 -- 172.21.15.151:0/2592036571 >> v1:172.21.15.151:6789/0 conn(0x7fafec0073b0 legacy=0x7fafec0cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7faff8919640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7faff8919640 1 monclient: found mon.noname-b 2023-11-02T08:36:32.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7faff8919640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7faff8919640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.319+0000 7faff8919640 1 -- 172.21.15.151:0/2592036571 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec0338e0 con 0x7fafec006fd0 2023-11-02T08:36:32.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fc4a49f0640 1 Processor -- start 2023-11-02T08:36:32.384 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fc4a49f0640 1 -- start start 2023-11-02T08:36:32.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fc4a49f0640 10 monclient: init 2023-11-02T08:36:32.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fc4a49f0640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fc4a49f0640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-11-02T08:36:32.385 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fc4a49f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4a00706b0 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:32.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fb72b7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb7240c53e0 0x7fb724038f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36272/0 (socket says 172.21.15.151:36272) 2023-11-02T08:36:32.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fb72b7fe640 1 -- 172.21.15.151:0/3377893532 learned_addr learned my addr 172.21.15.151:0/3377893532 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.386 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fb72b7fe640 10 monclient(hunting): get_auth_request con 0x7fb7240c53e0 auth_method 0 2023-11-02T08:36:32.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fb72b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fb72b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.320+0000 7fb72b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc4a006d830 0x7fc4a0141750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a006d830 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:32.389 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a0141c90 0x7fc4a013add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.389 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7f64937fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f649406f810 0x7f649413fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:42898/0 (socket says 172.21.15.151:42898) 2023-11-02T08:36:32.389 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7f64937fe640 1 -- 172.21.15.151:0/3682822413 learned_addr learned my addr 172.21.15.151:0/3682822413 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:32.390 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a0141c90 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.390 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a006fcb0 con 0x7fc4a00706b0 2023-11-02T08:36:32.390 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a006ff20 con 0x7fc4a006d830 2023-11-02T08:36:32.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a0141230 con 0x7fc4a0141c90 2023-11-02T08:36:32.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.391 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.321+0000 7fc4a49f0640 10 monclient(hunting): authenticate will time out at 1363.677072s 2023-11-02T08:36:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.323+0000 7f2f4affd640 1 --2- 172.21.15.151:0/3041811067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c0b75f0 0x7f2f4c0a8750 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009870 tx=0x7f2f4002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88d7a9d76a42b6b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.323+0000 7faff8919640 1 --2- 172.21.15.151:0/2592036571 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec006fd0 0x7fafec0355e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faff4079c10 tx=0x7faff40a5710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6058bcafc236044d server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.323+0000 7f5371d9b640 1 --2- 172.21.15.151:0/2004493105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f537413e3b0 0x7f53740ad840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f53740a8cf0 tx=0x7f536402c1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=118dfdfa3c02ed92 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.324+0000 7fb72b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.324+0000 7fb72b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.324+0000 7fb72b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.324+0000 7fc49ed9d640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a0141c90 0x7fc4a013add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.325+0000 7fb72bfff640 1 --2- 172.21.15.151:0/3377893532 >> v2:172.21.15.151:3300/0 conn(0x7fb7240cbb50 0x7fb7240d2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.326+0000 7fb72bfff640 10 monclient(hunting): get_auth_request con 0x7fb7240cbb50 auth_method 0 2023-11-02T08:36:32.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.326+0000 7fb72bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.326+0000 7fb72bfff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.326+0000 7fb72bfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.327+0000 7fb72b7fe640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2023-11-02T08:36:32.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.327+0000 7fb72b7fe640 1 -- 172.21.15.151:0/3377893532 >> v2:172.21.15.151:3300/0 conn(0x7fb7240cbb50 msgr2=0x7fb7240d2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.327+0000 7fb72b7fe640 1 --2- 172.21.15.151:0/3377893532 >> v2:172.21.15.151:3300/0 conn(0x7fb7240cbb50 0x7fb7240d2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.327+0000 7fb72b7fe640 1 -- 172.21.15.151:0/3377893532 >> v1:172.21.15.151:6789/0 conn(0x7fb724036360 legacy=0x7fb7240cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.327+0000 7fb72b7fe640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.327+0000 7fb72b7fe640 1 monclient: found mon.noname-c 2023-11-02T08:36:32.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.327+0000 7fb72b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.327+0000 7fb72b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.327+0000 7fb72b7fe640 1 -- 172.21.15.151:0/3377893532 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb724039470 con 0x7fb7240c53e0 2023-11-02T08:36:32.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 1 -- 172.21.15.151:0/2592036571 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40aa5d0 con 0x7fafec006fd0 2023-11-02T08:36:32.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:32.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 10 monclient: dump: 2023-11-02T08:36:32.400 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.400 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.400 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.400 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.401 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.401 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.401 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.401 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.401 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.402 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 1 -- 172.21.15.151:0/2592036571 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff40a8050 con 0x7fafec006fd0 2023-11-02T08:36:32.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 1 -- 172.21.15.151:0/2592036571 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40aaa30 con 0x7fafec006fd0 2023-11-02T08:36:32.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:32.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faff37fe640 10 monclient: dump: 2023-11-02T08:36:32.404 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.404 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.405 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.405 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.405 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.406 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.406 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.406 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.406 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.407 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.407 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faffab7d640 5 monclient: authenticate success, global_id 4265 2023-11-02T08:36:32.407 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faffab7d640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:32.407 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.328+0000 7faffab7d640 4 set_mon_vals no callback set 2023-11-02T08:36:32.407 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 10 monclient: shutdown 2023-11-02T08:36:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 1 -- 172.21.15.151:0/2592036571 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec006fd0 msgr2=0x7fafec0355e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 1 --2- 172.21.15.151:0/2592036571 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec006fd0 0x7fafec0355e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faff4079c10 tx=0x7faff40a5710 comp rx=0 tx=0).stop 2023-11-02T08:36:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 1 -- 172.21.15.151:0/2592036571 shutdown_connections 2023-11-02T08:36:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 1 --2- 172.21.15.151:0/2592036571 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec006fd0 0x7fafec0355e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 1 -- 172.21.15.151:0/2592036571 >> 172.21.15.151:0/2592036571 conn(0x7fafec0061f0 msgr2=0x7fafec006600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 1 -- 172.21.15.151:0/2592036571 shutdown_connections 2023-11-02T08:36:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7fc49ed9d640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a0141c90 0x7fc4a013add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54816/0 (socket says 172.21.15.151:54816) 2023-11-02T08:36:32.409 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7fc49ed9d640 1 -- 172.21.15.151:0/613672604 learned_addr learned my addr 172.21.15.151:0/613672604 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7fc49ed9d640 10 monclient(hunting): get_auth_request con 0x7fc4a0141c90 auth_method 0 2023-11-02T08:36:32.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 1 -- 172.21.15.151:0/2592036571 wait complete. 2023-11-02T08:36:32.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7fc49ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7fc49ed9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7fc49ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.411 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.412 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.329+0000 7faffab7d640 10 monclient: monmap: 2023-11-02T08:36:32.412 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.412 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:32.412 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:32.412 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:32.413 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.413 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.413 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:32.413 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:32.413 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:32.414 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7faffab7d640 1 librados: starting msgr at 2023-11-02T08:36:32.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7faffab7d640 1 librados: starting objecter 2023-11-02T08:36:32.414 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7fc49ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7fc49ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7fc49ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.415 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7faffab7d640 1 Processor -- start 2023-11-02T08:36:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7faffab7d640 1 -- start start 2023-11-02T08:36:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7faffab7d640 1 librados: setting wanted keys 2023-11-02T08:36:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7faffab7d640 1 librados: calling monclient init 2023-11-02T08:36:32.416 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.330+0000 7faffab7d640 10 monclient: init 2023-11-02T08:36:32.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7fb72b7fe640 1 --2- 172.21.15.151:0/3377893532 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7240c53e0 0x7fb724038f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb72c079c10 tx=0x7fb72c0a5580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47056bb84db19aa2 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:36:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafec006fd0 0x7fafec0c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient(hunting): picked mon.noname-a con 0x7fafec006fd0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:32.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafec030ae0 0x7fafec02ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient(hunting): picked mon.noname-b con 0x7fafec030ae0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:32.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec02d5e0 0x7fafec0c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient(hunting): picked mon.noname-c con 0x7fafec02d5e0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec02e8b0 con 0x7fafec006fd0 2023-11-02T08:36:32.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0c6c80 con 0x7fafec030ae0 2023-11-02T08:36:32.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0c78c0 con 0x7fafec02d5e0 2023-11-02T08:36:32.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.331+0000 7faffab7d640 10 monclient(hunting): authenticate will time out at 1363.687353s 2023-11-02T08:36:32.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.332+0000 7fc49dd9b640 1 --2- 172.21.15.151:0/613672604 >> v2:172.21.15.178:3300/0 conn(0x7fc4a006d830 0x7fc4a0141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7fc49dd9b640 10 monclient(hunting): get_auth_request con 0x7fc4a006d830 auth_method 0 2023-11-02T08:36:32.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7fc49dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7fc49dd9b640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7fc49dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551938443 0 0) 0x7f6494070bf0 con 0x7f649406f810 2023-11-02T08:36:32.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7f64927fc640 10 monclient(hunting): _negotiate 2023-11-02T08:36:32.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7f64927fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7f64927fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7f64927fc640 10 monclient(hunting): my global_id is 4339 2023-11-02T08:36:32.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.333+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c003520 con 0x7f649406f810 2023-11-02T08:36:32.424 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 1 -- 172.21.15.151:0/1971656940 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc0aa510 con 0x7f4ed40357e0 2023-11-02T08:36:32.425 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.425 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:32.425 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 10 monclient: dump: 2023-11-02T08:36:32.425 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.425 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.426 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.426 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.426 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.426 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.426 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.427 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.427 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.427 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.427 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.427 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 1 -- 172.21.15.151:0/1971656940 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4edc0a8050 con 0x7f4ed40357e0 2023-11-02T08:36:32.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 1 -- 172.21.15.151:0/1971656940 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc0aa730 con 0x7f4ed40357e0 2023-11-02T08:36:32.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:32.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4edaffd640 10 monclient: dump: 2023-11-02T08:36:32.429 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.429 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.429 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.429 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.430 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.430 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.430 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.430 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.430 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.431 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4ee15d8640 5 monclient: authenticate success, global_id 4284 2023-11-02T08:36:32.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4ee15d8640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:32.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.334+0000 7f4ee15d8640 4 set_mon_vals no callback set 2023-11-02T08:36:32.431 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7f4ee15d8640 10 monclient: shutdown 2023-11-02T08:36:32.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7fc49ed9d640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2023-11-02T08:36:32.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7fc49ed9d640 1 -- 172.21.15.151:0/613672604 >> v2:172.21.15.178:3300/0 conn(0x7fc4a006d830 msgr2=0x7fc4a0141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7fc49ed9d640 1 --2- 172.21.15.151:0/613672604 >> v2:172.21.15.178:3300/0 conn(0x7fc4a006d830 0x7fc4a0141750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.432 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7fc49ed9d640 1 -- 172.21.15.151:0/613672604 >> v1:172.21.15.151:6789/0 conn(0x7fc4a00706b0 legacy=0x7fc4a0070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7f4ee15d8640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7fc49ed9d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7fc49ed9d640 1 monclient: found mon.noname-b 2023-11-02T08:36:32.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7fc49ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7fc49ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7fc49ed9d640 1 -- 172.21.15.151:0/613672604 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a013b310 con 0x7fc4a0141c90 2023-11-02T08:36:32.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7f4ee15d8640 1 -- 172.21.15.151:0/1971656940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ed40357e0 msgr2=0x7f4ed40c8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7f4ee15d8640 1 --2- 172.21.15.151:0/1971656940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ed40357e0 0x7f4ed40c8500 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4edc068260 tx=0x7f4edc0a5580 comp rx=0 tx=0).stop 2023-11-02T08:36:32.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7f4ee15d8640 1 -- 172.21.15.151:0/1971656940 shutdown_connections 2023-11-02T08:36:32.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7f4ee15d8640 1 --2- 172.21.15.151:0/1971656940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ed40357e0 0x7f4ed40c8500 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7f4ee15d8640 1 -- 172.21.15.151:0/1971656940 >> 172.21.15.151:0/1971656940 conn(0x7f4ed4006690 msgr2=0x7f4ed4006aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7f4ee15d8640 1 -- 172.21.15.151:0/1971656940 shutdown_connections 2023-11-02T08:36:32.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.335+0000 7f4ee15d8640 1 -- 172.21.15.151:0/1971656940 wait complete. 2023-11-02T08:36:32.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.336+0000 7f4ee15d8640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.336+0000 7f4ee15d8640 10 monclient: monmap: 2023-11-02T08:36:32.437 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.438 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:32.438 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:32.438 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:32.438 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.438 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.439 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:32.439 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:32.439 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:32.439 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.336+0000 7f4ee15d8640 1 librados: starting msgr at 2023-11-02T08:36:32.439 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.336+0000 7f4ee15d8640 1 librados: starting objecter 2023-11-02T08:36:32.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.336+0000 7f4ee15d8640 1 Processor -- start 2023-11-02T08:36:32.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.336+0000 7f4ee15d8640 1 -- start start 2023-11-02T08:36:32.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.336+0000 7f4ee15d8640 1 librados: setting wanted keys 2023-11-02T08:36:32.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.336+0000 7f4ee15d8640 1 librados: calling monclient init 2023-11-02T08:36:32.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.336+0000 7f4ee15d8640 10 monclient: init 2023-11-02T08:36:32.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 1 -- 172.21.15.151:0/3041811067 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f400317c0 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:32.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 10 monclient: dump: 2023-11-02T08:36:32.442 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.442 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.442 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.442 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.443 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.443 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.443 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.443 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.443 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.444 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.444 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.444 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f6492ffd640 1 --2- 172.21.15.151:0/3682822413 >> v2:172.21.15.151:3300/0 conn(0x7f64940af750 0x7f649406fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 1 -- 172.21.15.151:0/3041811067 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f40031920 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 1 -- 172.21.15.151:0/3041811067 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f40031cb0 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:32.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f49ffb640 10 monclient: dump: 2023-11-02T08:36:32.446 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.446 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.447 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.447 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.447 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.447 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.447 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.448 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.448 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.448 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f51841640 5 monclient: authenticate success, global_id 4262 2023-11-02T08:36:32.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f51841640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:32.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f2f51841640 4 set_mon_vals no callback set 2023-11-02T08:36:32.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f4ee15d8640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f4ee15d8640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-02T08:36:32.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f4ee15d8640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ed40357e0 0x7f4ed40c2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f4ee15d8640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed40357e0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f4ee15d8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed4038f10 0x7f4ed402fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f4ee15d8640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed4038f10 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:32.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415718485 0 0) 0x7f649413f630 con 0x7f649413c6e0 2023-11-02T08:36:32.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.337+0000 7f4ee15d8640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ed40d31b0 0x7f4ed40c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f4ee15d8640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed40d31b0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:32.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f4ee15d8640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f4ee15d8640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40c2e90 con 0x7f4ed40d31b0 2023-11-02T08:36:32.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f4ee15d8640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f4ee15d8640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed4030110 con 0x7f4ed4038f10 2023-11-02T08:36:32.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f4ee15d8640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f4ee15d8640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed415fdb0 con 0x7f4ed40357e0 2023-11-02T08:36:32.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f4ee15d8640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f4ee15d8640 10 monclient(hunting): authenticate will time out at 1363.693956s 2023-11-02T08:36:32.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f64927fc640 10 monclient(hunting): _negotiate 2023-11-02T08:36:32.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f64927fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.454 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f64927fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f64927fc640 10 monclient(hunting): my global_id is 4293 2023-11-02T08:36:32.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6494070bf0 con 0x7f649413c6e0 2023-11-02T08:36:32.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 10 monclient: shutdown 2023-11-02T08:36:32.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 1 -- 172.21.15.151:0/3041811067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c0b75f0 msgr2=0x7f2f4c0a8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 1 --2- 172.21.15.151:0/3041811067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c0b75f0 0x7f2f4c0a8750 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009870 tx=0x7f2f4002b860 comp rx=0 tx=0).stop 2023-11-02T08:36:32.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 1 -- 172.21.15.151:0/3041811067 shutdown_connections 2023-11-02T08:36:32.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 1 --2- 172.21.15.151:0/3041811067 >> v2:172.21.15.151:3300/0 conn(0x7f2f4c0a8c90 0x7f2f4c13e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 1 --2- 172.21.15.151:0/3041811067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c0b75f0 0x7f2f4c0a8750 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 1 -- 172.21.15.151:0/3041811067 >> 172.21.15.151:0/3041811067 conn(0x7f2f4c065d50 msgr2=0x7f2f4c066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 1 -- 172.21.15.151:0/3041811067 shutdown_connections 2023-11-02T08:36:32.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7f2f51841640 1 -- 172.21.15.151:0/3041811067 wait complete. 2023-11-02T08:36:32.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.338+0000 7faff911a640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec02d5e0 0x7fafec0c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7f6492ffd640 10 monclient(hunting): get_auth_request con 0x7f64940af750 auth_method 0 2023-11-02T08:36:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7f6492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7f2f51841640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7f6492ffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7f6492ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7f2f51841640 10 monclient: monmap: 2023-11-02T08:36:32.464 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.464 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:32.464 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:32.464 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:32.466 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7faff911a640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec02d5e0 0x7fafec0c5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54824/0 (socket says 172.21.15.151:54824) 2023-11-02T08:36:32.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7faff911a640 1 -- 172.21.15.151:0/3745728382 learned_addr learned my addr 172.21.15.151:0/3745728382 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7faff911a640 10 monclient(hunting): get_auth_request con 0x7fafec02d5e0 auth_method 0 2023-11-02T08:36:32.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7faff911a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7faff911a640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7faff911a640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7f2f51841640 1 librados: starting msgr at 2023-11-02T08:36:32.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.339+0000 7f2f51841640 1 librados: starting objecter 2023-11-02T08:36:32.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f2f51841640 1 Processor -- start 2023-11-02T08:36:32.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f2f51841640 1 -- start start 2023-11-02T08:36:32.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f2f51841640 1 librados: setting wanted keys 2023-11-02T08:36:32.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f2f51841640 1 librados: calling monclient init 2023-11-02T08:36:32.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f2f51841640 10 monclient: init 2023-11-02T08:36:32.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f6492ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f6492ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f6492ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7faff911a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f2f51841640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f2f51841640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:36:32.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7f2f51841640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c0a8c90 0x7f2f4c1cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7faff911a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.340+0000 7faff911a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 10 monclient(hunting): picked mon.noname-a con 0x7f2f4c0a8c90 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:32.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f4c0b75f0 0x7f2f4c1d61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 10 monclient(hunting): picked mon.noname-b con 0x7f2f4c0b75f0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:32.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c13ec50 0x7f2f4c1d6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 10 monclient(hunting): picked mon.noname-c con 0x7f2f4c13ec50 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c140520 con 0x7f2f4c0a8c90 2023-11-02T08:36:32.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c13d7a0 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c13d960 con 0x7f2f4c13ec50 2023-11-02T08:36:32.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.341+0000 7f2f51841640 10 monclient(hunting): authenticate will time out at 1363.697057s 2023-11-02T08:36:32.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 1 -- 172.21.15.151:0/3377893532 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0aa5d0 con 0x7fb7240c53e0 2023-11-02T08:36:32.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:32.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1739701122 0 0) 0x7f647c003520 con 0x7f649406f810 2023-11-02T08:36:32.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 >> v2:172.21.15.151:3300/0 conn(0x7f64940af750 msgr2=0x7f649406fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 --2- 172.21.15.151:0/3682822413 >> v2:172.21.15.151:3300/0 conn(0x7f64940af750 0x7f649406fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.479 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 10 monclient: dump: 2023-11-02T08:36:32.479 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.479 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.479 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.479 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.480 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.480 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.480 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.480 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.480 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.481 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 >> v1:172.21.15.151:6789/0 conn(0x7f649413c6e0 legacy=0x7f649413cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 1 -- 172.21.15.151:0/3377893532 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c0a8050 con 0x7fb7240c53e0 2023-11-02T08:36:32.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 monclient: found mon.noname-c-legacy 2023-11-02T08:36:32.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.178:6789/0 2023-11-02T08:36:32.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6494071a60 con 0x7f649406f810 2023-11-02T08:36:32.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 650167734 0 0) 0x7f6484001670 con 0x7f649406f810 2023-11-02T08:36:32.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-11-02T08:36:32.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient: dump: 2023-11-02T08:36:32.485 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.486 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.486 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.486 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.486 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.486 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.486 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.487 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.487 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.487 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] but i'm connected to v1:172.21.15.178:6789/0, reconnecting 2023-11-02T08:36:32.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 >> v1:172.21.15.178:6789/0 conn(0x7f649406f810 legacy=0x7f649413fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:36:32.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 --2- 172.21.15.151:0/3682822413 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c003b50 0x7f647c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient(hunting): picked mon.a con 0x7f647c003b50 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:32.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 --2- 172.21.15.151:0/3682822413 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 0x7f647c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f4edb7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed4038f10 0x7f4ed402fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient(hunting): picked mon.c con 0x7f647c0047c0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 --2- 172.21.15.151:0/3682822413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647c005180 0x7f647c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 1 -- 172.21.15.151:0/3377893532 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0aa810 con 0x7fb7240c53e0 2023-11-02T08:36:32.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:32.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb72a7fc640 10 monclient: dump: 2023-11-02T08:36:32.491 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.491 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.492 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.492 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.492 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.492 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.492 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.493 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.493 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.493 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb731ff6640 5 monclient: authenticate success, global_id 4336 2023-11-02T08:36:32.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb731ff6640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:32.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7fb731ff6640 4 set_mon_vals no callback set 2023-11-02T08:36:32.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient(hunting): picked mon.b con 0x7f647c005180 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:32.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6494071780 con 0x7f647c003b50 2023-11-02T08:36:32.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c00c050 con 0x7f647c005180 2023-11-02T08:36:32.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c00c210 con 0x7f647c0047c0 2023-11-02T08:36:32.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.343+0000 7f64927fc640 10 monclient(hunting): _finish_auth 0 2023-11-02T08:36:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2023-11-02T08:36:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 1 -- 172.21.15.151:0/3745728382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafec030ae0 msgr2=0x7fafec02ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafec030ae0 0x7fafec02ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 1 -- 172.21.15.151:0/3745728382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafec006fd0 msgr2=0x7fafec0c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafec006fd0 0x7fafec0c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 1 monclient: found mon.noname-c 2023-11-02T08:36:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f2f4a7fc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f4c0b75f0 0x7f2f4c1d61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 10 monclient: shutdown 2023-11-02T08:36:32.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 1 -- 172.21.15.151:0/3377893532 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7240c53e0 msgr2=0x7fb724038f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 1 --2- 172.21.15.151:0/3377893532 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7240c53e0 0x7fb724038f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb72c079c10 tx=0x7fb72c0a5580 comp rx=0 tx=0).stop 2023-11-02T08:36:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f4edb7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed4038f10 0x7f4ed402fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36304/0 (socket says 172.21.15.151:36304) 2023-11-02T08:36:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f4edb7fe640 1 -- 172.21.15.151:0/2448039729 learned_addr learned my addr 172.21.15.151:0/2448039729 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f4edb7fe640 10 monclient(hunting): get_auth_request con 0x7f4ed4038f10 auth_method 0 2023-11-02T08:36:32.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f4edb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f4edb7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f4edb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 1 -- 172.21.15.151:0/3377893532 shutdown_connections 2023-11-02T08:36:32.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 1 --2- 172.21.15.151:0/3377893532 >> v2:172.21.15.151:3300/0 conn(0x7fb7240cbb50 0x7fb7240d2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 1 --2- 172.21.15.151:0/3377893532 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7240c53e0 0x7fb724038f30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 1 -- 172.21.15.151:0/3377893532 >> 172.21.15.151:0/3377893532 conn(0x7fb724006720 msgr2=0x7fb724006b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7faff911a640 1 -- 172.21.15.151:0/3745728382 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec0ca6c0 con 0x7fafec02d5e0 2023-11-02T08:36:32.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 1 -- 172.21.15.151:0/3377893532 shutdown_connections 2023-11-02T08:36:32.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f2f4a7fc640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f4c0b75f0 0x7f2f4c1d61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36318/0 (socket says 172.21.15.151:36318) 2023-11-02T08:36:32.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f2f4a7fc640 1 -- 172.21.15.151:0/54319016 learned_addr learned my addr 172.21.15.151:0/54319016 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f2f4a7fc640 10 monclient(hunting): get_auth_request con 0x7f2f4c0b75f0 auth_method 0 2023-11-02T08:36:32.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f2f4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f2f4a7fc640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7f2f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.344+0000 7fb731ff6640 1 -- 172.21.15.151:0/3377893532 wait complete. 2023-11-02T08:36:32.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7fb731ff6640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7fb731ff6640 10 monclient: monmap: 2023-11-02T08:36:32.506 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.506 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:32.506 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:32.506 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:32.507 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.507 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.507 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:32.507 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:32.507 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:32.508 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 1 -- 172.21.15.151:0/2004493105 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364034440 con 0x7f537413e3b0 2023-11-02T08:36:32.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:32.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 10 monclient: dump: 2023-11-02T08:36:32.510 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.510 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.510 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.510 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.510 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.511 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.511 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.511 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.511 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.511 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 1 -- 172.21.15.151:0/2004493105 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53640345a0 con 0x7f537413e3b0 2023-11-02T08:36:32.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.512 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 1 -- 172.21.15.151:0/2004493105 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364034930 con 0x7f537413e3b0 2023-11-02T08:36:32.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:32.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537159a640 10 monclient: dump: 2023-11-02T08:36:32.513 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.513 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.514 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.514 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.514 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.514 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.514 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.514 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.515 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.515 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537359e640 5 monclient: authenticate success, global_id 4330 2023-11-02T08:36:32.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537359e640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:32.515 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f537359e640 4 set_mon_vals no callback set 2023-11-02T08:36:32.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7fb731ff6640 1 librados: starting msgr at 2023-11-02T08:36:32.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7fb731ff6640 1 librados: starting objecter 2023-11-02T08:36:32.516 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f4edb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f4edb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7f4edb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.517 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.345+0000 7fb731ff6640 1 Processor -- start 2023-11-02T08:36:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 1 -- start start 2023-11-02T08:36:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 1 librados: setting wanted keys 2023-11-02T08:36:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 1 librados: calling monclient init 2023-11-02T08:36:32.518 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient: init 2023-11-02T08:36:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f2f4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f2f4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f2f4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f537359e640 10 monclient: shutdown 2023-11-02T08:36:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fd85a527640 10 monclient: get_monmap_and_config 2023-11-02T08:36:32.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f537359e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f537359e640 1 -- 172.21.15.151:0/2004493105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f537413e3b0 msgr2=0x7f53740ad840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f537359e640 1 --2- 172.21.15.151:0/2004493105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f537413e3b0 0x7f53740ad840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f53740a8cf0 tx=0x7f536402c1b0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.521 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fd85a527640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fd85a527640 10 monclient: monmap: 2023-11-02T08:36:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:32.347575+0000 2023-11-02T08:36:32.522 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:32.347575+0000 2023-11-02T08:36:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:32.523 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:32.524 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:32.524 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:32.524 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-02T08:36:32.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb724036360 0x7fb7240a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.525 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f537359e640 1 -- 172.21.15.151:0/2004493105 shutdown_connections 2023-11-02T08:36:32.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f537359e640 1 --2- 172.21.15.151:0/2004493105 >> v2:172.21.15.151:3300/0 conn(0x7f53740add80 0x7f53740ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f537359e640 1 --2- 172.21.15.151:0/2004493105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f537413e3b0 0x7f53740ad840 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient(hunting): picked mon.noname-c con 0x7fb724036360 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.526 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7240c53e0 0x7fb7240c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient(hunting): picked mon.noname-a con 0x7fb7240c53e0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:32.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7240cbb50 0x7fb7240c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient(hunting): picked mon.noname-b con 0x7fb7240cbb50 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:32.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72402e5f0 con 0x7fb7240c53e0 2023-11-02T08:36:32.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f537359e640 1 -- 172.21.15.151:0/2004493105 >> 172.21.15.151:0/2004493105 conn(0x7f53740b2ae0 msgr2=0x7f53740b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7240a5f60 con 0x7fb7240cbb50 2023-11-02T08:36:32.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7240a7790 con 0x7fb724036360 2023-11-02T08:36:32.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7fb731ff6640 10 monclient(hunting): authenticate will time out at 1363.702680s 2023-11-02T08:36:32.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.346+0000 7f537359e640 1 -- 172.21.15.151:0/2004493105 shutdown_connections 2023-11-02T08:36:32.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.347+0000 7f537359e640 1 -- 172.21.15.151:0/2004493105 wait complete. 2023-11-02T08:36:32.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.347+0000 7fd85a527640 1 Processor -- start 2023-11-02T08:36:32.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.347+0000 7fd85a527640 1 -- start start 2023-11-02T08:36:32.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.347+0000 7fd85a527640 10 monclient: init 2023-11-02T08:36:32.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.347+0000 7f537359e640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.347+0000 7f537359e640 10 monclient: monmap: 2023-11-02T08:36:32.532 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.532 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:32.532 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:32.532 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:32.532 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.533 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.533 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:32.533 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:32.533 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:32.533 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.347+0000 7faff911a640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec02d5e0 0x7fafec0c5920 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fafe800b490 tx=0x7fafe800b950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.347+0000 7faff17fa640 10 client.?.objecter ms_handle_connect 0x7fafec02d5e0 2023-11-02T08:36:32.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.347+0000 7faff17fa640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:32.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f537359e640 1 librados: starting msgr at 2023-11-02T08:36:32.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f537359e640 1 librados: starting objecter 2023-11-02T08:36:32.535 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edbfff640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ed40357e0 0x7f4ed40c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f537359e640 1 Processor -- start 2023-11-02T08:36:32.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f537359e640 1 -- start start 2023-11-02T08:36:32.536 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f537359e640 1 librados: setting wanted keys 2023-11-02T08:36:32.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-11-02T08:36:32.537 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8540b4570 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:32.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd8540b0ed0 0x7fd854144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.538 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f537359e640 1 librados: calling monclient init 2023-11-02T08:36:32.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 10 monclient(hunting): picked mon.noname-c con 0x7fd8540b0ed0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:32.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd85413e520 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:32.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540a9780 con 0x7fd8540b4570 2023-11-02T08:36:32.539 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540aa1d0 con 0x7fd85413e520 2023-11-02T08:36:32.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8540b4950 con 0x7fd8540b0ed0 2023-11-02T08:36:32.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.540 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7fd85a527640 10 monclient(hunting): authenticate will time out at 1363.704566s 2023-11-02T08:36:32.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f537359e640 10 monclient: init 2023-11-02T08:36:32.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2023-11-02T08:36:32.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 1 -- 172.21.15.151:0/2448039729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ed40357e0 msgr2=0x7f4ed40c2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ed40357e0 0x7f4ed40c2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 1 -- 172.21.15.151:0/2448039729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ed40d31b0 msgr2=0x7f4ed40c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ed40d31b0 0x7f4ed40c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.542 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 1 monclient: found mon.noname-b 2023-11-02T08:36:32.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.543 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.348+0000 7f4edb7fe640 1 -- 172.21.15.151:0/2448039729 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc0669d0 con 0x7f4ed4038f10 2023-11-02T08:36:32.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:36:32.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53740a8720 0x7f537413a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient(hunting): picked mon.noname-a con 0x7f53740a8720 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:32.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53740add80 0x7f537406cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient(hunting): picked mon.noname-b con 0x7f53740add80 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:32.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f537413e3b0 0x7f53741404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient(hunting): picked mon.noname-c con 0x7f537413e3b0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537413fe70 con 0x7f53740a8720 2023-11-02T08:36:32.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537406df70 con 0x7f53740add80 2023-11-02T08:36:32.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537413afa0 con 0x7f537413e3b0 2023-11-02T08:36:32.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.349+0000 7f537359e640 10 monclient(hunting): authenticate will time out at 1363.705151s 2023-11-02T08:36:32.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2023-11-02T08:36:32.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 1 -- 172.21.15.151:0/54319016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c13ec50 msgr2=0x7f2f4c1d6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c13ec50 0x7f2f4c1d6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 1 -- 172.21.15.151:0/54319016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c0a8c90 msgr2=0x7f2f4c1cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c0a8c90 0x7f2f4c1cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 1 monclient: found mon.noname-b 2023-11-02T08:36:32.549 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.550 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.352+0000 7f2f4a7fc640 1 -- 172.21.15.151:0/54319016 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f400094d0 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 1 -- 172.21.15.151:0/3745728382 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8011830 con 0x7fafec02d5e0 2023-11-02T08:36:32.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.551 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:32.552 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 10 monclient: dump: 2023-11-02T08:36:32.552 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.552 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.552 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.552 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.553 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.553 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.553 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.553 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.553 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.554 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.554 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faffab7d640 5 monclient: authenticate success, global_id 4271 2023-11-02T08:36:32.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 1 -- 172.21.15.151:0/3745728382 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe80119b0 con 0x7fafec02d5e0 2023-11-02T08:36:32.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 1 -- 172.21.15.151:0/3745728382 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8011d60 con 0x7fafec02d5e0 2023-11-02T08:36:32.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:32.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff17fa640 10 monclient: dump: 2023-11-02T08:36:32.556 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.556 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.556 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.556 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.556 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.557 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.557 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.557 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.557 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.557 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faff2ffd640 4 set_mon_vals no callback set 2023-11-02T08:36:32.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faffab7d640 10 monclient: _renew_subs 2023-11-02T08:36:32.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faffab7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec0ca970 con 0x7fafec02d5e0 2023-11-02T08:36:32.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faffab7d640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:32.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faffab7d640 10 monclient: _renew_subs 2023-11-02T08:36:32.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faffab7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec02f3f0 con 0x7fafec02d5e0 2023-11-02T08:36:32.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.354+0000 7faffab7d640 1 librados: init done 2023-11-02T08:36:32.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.355+0000 7f5372d9d640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f537413e3b0 0x7f53741404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.355+0000 7faffab7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.355+0000 7faffab7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.355+0000 7faffab7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.355+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec0ca3e0 con 0x7fafec02d5e0 2023-11-02T08:36:32.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7fb72b7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb724036360 0x7fb7240a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.561 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7f6492ffd640 1 --2- 172.21.15.151:0/3682822413 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 0x7f647c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7f2f4a7fc640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f4c0b75f0 0x7f2f4c1d61c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00b4a0 tx=0x7f2f3c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7f2f33fff640 10 client.?.objecter ms_handle_connect 0x7f2f4c0b75f0 2023-11-02T08:36:32.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7f2f33fff640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:32.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7fb72b7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb724036360 0x7fb7240a6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54862/0 (socket says 172.21.15.151:54862) 2023-11-02T08:36:32.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7fb72b7fe640 1 -- 172.21.15.151:0/2787737699 learned_addr learned my addr 172.21.15.151:0/2787737699 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7fb72b7fe640 10 monclient(hunting): get_auth_request con 0x7fb724036360 auth_method 0 2023-11-02T08:36:32.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7fb72b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7fb72b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.356+0000 7fb72b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f5372d9d640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f537413e3b0 0x7f53741404b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54876/0 (socket says 172.21.15.151:54876) 2023-11-02T08:36:32.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f5372d9d640 1 -- 172.21.15.151:0/2425773961 learned_addr learned my addr 172.21.15.151:0/2425773961 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f5372d9d640 10 monclient(hunting): get_auth_request con 0x7f537413e3b0 auth_method 0 2023-11-02T08:36:32.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f5372d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f5372d9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f5372d9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f6492ffd640 10 monclient(hunting): get_auth_request con 0x7f647c0047c0 auth_method 0 2023-11-02T08:36:32.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f6492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f6492ffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.357+0000 7f6492ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:36:32.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7f6492ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7f6492ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7f6492ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-02T08:36:32.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7fb72affd640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7240c53e0 0x7fb7240c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7f4edb7fe640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed4038f10 0x7f4ed402fbd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800c410 tx=0x7f4ec800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7f4ed8ff9640 10 client.?.objecter ms_handle_connect 0x7f4ed4038f10 2023-11-02T08:36:32.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7f4ed8ff9640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:32.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7fb72b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.569 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7fb72b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.358+0000 7fb72b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.359+0000 7f5372d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.570 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.359+0000 7f5372d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.359+0000 7f5372d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.359+0000 7fb72affd640 10 monclient(hunting): get_auth_request con 0x7fb7240c53e0 auth_method 0 2023-11-02T08:36:32.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.359+0000 7fb72affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.359+0000 7fb72affd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.359+0000 7fb72affd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.360+0000 7fb72affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.360+0000 7fb72affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.360+0000 7fb72affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 10 monclient(hunting): handle_auth_done global_id 4339 payload 293 2023-11-02T08:36:32.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 1 -- 172.21.15.151:0/3682822413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647c005180 msgr2=0x7f647c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 1 --2- 172.21.15.151:0/3682822413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647c005180 0x7f647c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 1 -- 172.21.15.151:0/3682822413 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c003b50 msgr2=0x7f647c003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:32.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 1 --2- 172.21.15.151:0/3682822413 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f647c003b50 0x7f647c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 1 monclient: found mon.c 2023-11-02T08:36:32.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 1 -- 172.21.15.151:0/3682822413 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c003520 con 0x7f647c0047c0 2023-11-02T08:36:32.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.575 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6498e1b640 5 monclient: authenticate success, global_id 4339 2023-11-02T08:36:32.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6498e1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:32.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7f6493fff640 1 -- 172.21.15.151:0/3682822413 reap_dead start 2023-11-02T08:36:32.576 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.361+0000 7fb72bfff640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7240cbb50 0x7fb7240c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.362+0000 7f5371d9b640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53740add80 0x7f537406cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.577 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2023-11-02T08:36:32.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 1 -- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7240cbb50 msgr2=0x7fb7240c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7240cbb50 0x7fb7240c4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 1 -- 172.21.15.151:0/2787737699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7240c53e0 msgr2=0x7fb7240c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.578 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7240c53e0 0x7fb7240c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 1 monclient: found mon.noname-c 2023-11-02T08:36:32.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.579 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.363+0000 7fb72b7fe640 1 -- 172.21.15.151:0/2787737699 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0669d0 con 0x7fb724036360 2023-11-02T08:36:32.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.365+0000 7fd853fff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd8540b4570 0x7fd8540a9070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:34142/0 (socket says 172.21.15.151:34142) 2023-11-02T08:36:32.580 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.365+0000 7fd853fff640 1 -- 172.21.15.151:0/3321479830 learned_addr learned my addr 172.21.15.151:0/3321479830 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:32.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.365+0000 7f5371d9b640 10 monclient(hunting): get_auth_request con 0x7f53740add80 auth_method 0 2023-11-02T08:36:32.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.365+0000 7f5371d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.365+0000 7f5371d9b640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.581 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.365+0000 7f5371d9b640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2023-11-02T08:36:32.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 1 -- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53740add80 msgr2=0x7f537406cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53740add80 0x7f537406cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 1 -- 172.21.15.151:0/2425773961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53740a8720 msgr2=0x7f537413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.582 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53740a8720 0x7f537413a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 1 monclient: found mon.noname-c 2023-11-02T08:36:32.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.583 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.366+0000 7f5372d9d640 1 -- 172.21.15.151:0/2425773961 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364009d60 con 0x7f537413e3b0 2023-11-02T08:36:32.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.369+0000 7fd84b7fe640 1 --2- 172.21.15.151:0/3321479830 >> v2:172.21.15.178:3300/0 conn(0x7fd8540b0ed0 0x7fd854144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.370+0000 7fd84b7fe640 10 monclient(hunting): get_auth_request con 0x7fd8540b0ed0 auth_method 0 2023-11-02T08:36:32.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.370+0000 7fd84b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.584 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.370+0000 7fd84b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.370+0000 7fd84b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.371+0000 7fb72b7fe640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb724036360 0x7fb7240a6390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0727b0 tx=0x7fb72c068a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.585 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.371+0000 7fb717fff640 10 client.?.objecter ms_handle_connect 0x7fb724036360 2023-11-02T08:36:32.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.371+0000 7fb717fff640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:32.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.371+0000 7fd84b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.371+0000 7fd84b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.371+0000 7fd84b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.371+0000 7f6492ffd640 1 --2- 172.21.15.151:0/3682822413 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 0x7f647c004c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6488009a70 tx=0x7f647c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eb7d1913160cc31 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f5372d9d640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f537413e3b0 0x7f53741404b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f536c00b970 tx=0x7f536c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f5362ffd640 10 client.?.objecter ms_handle_connect 0x7f537413e3b0 2023-11-02T08:36:32.588 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f5362ffd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:32.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 1 -- 172.21.15.151:0/54319016 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c010610 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:32.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 10 monclient: dump: 2023-11-02T08:36:32.590 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.590 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.590 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.590 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.590 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.591 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.591 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.591 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.591 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.591 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 1 -- 172.21.15.151:0/54319016 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c010770 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.592 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 1 -- 172.21.15.151:0/54319016 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c010b20 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f33fff640 10 monclient: dump: 2023-11-02T08:36:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.593 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.594 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.594 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.594 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.594 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.594 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.595 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.595 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.595 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f497fa640 4 set_mon_vals no callback set 2023-11-02T08:36:32.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f51841640 5 monclient: authenticate success, global_id 4348 2023-11-02T08:36:32.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f51841640 10 monclient: _renew_subs 2023-11-02T08:36:32.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f51841640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4c1d6c40 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f51841640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:32.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f51841640 10 monclient: _renew_subs 2023-11-02T08:36:32.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f51841640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4c13f6f0 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.372+0000 7f2f51841640 1 librados: init done 2023-11-02T08:36:32.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.373+0000 7f2f51841640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.373+0000 7f2f51841640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.373+0000 7f2f51841640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.373+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4c140760 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.373+0000 7fd8537fe640 1 -- 172.21.15.151:0/3321479830 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055087043 0 0) 0x7fd8540a9780 con 0x7fd8540b4570 2023-11-02T08:36:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.374+0000 7fd8537fe640 10 monclient(hunting): _negotiate 2023-11-02T08:36:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.374+0000 7fd8537fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.374+0000 7fd8537fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.374+0000 7fd8537fe640 10 monclient(hunting): my global_id is 4302 2023-11-02T08:36:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.374+0000 7fd8537fe640 1 -- 172.21.15.151:0/3321479830 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd840003520 con 0x7fd8540b4570 2023-11-02T08:36:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.376+0000 7fd84b7fe640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2023-11-02T08:36:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.376+0000 7fd84b7fe640 1 -- 172.21.15.151:0/3321479830 >> v1:172.21.15.178:6789/0 conn(0x7fd85413e520 legacy=0x7fd85413e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.376+0000 7fd84b7fe640 1 -- 172.21.15.151:0/3321479830 >> v1:172.21.15.151:6789/0 conn(0x7fd8540b4570 legacy=0x7fd8540a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.376+0000 7fd84b7fe640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.376+0000 7fd84b7fe640 1 monclient: found mon.noname-c 2023-11-02T08:36:32.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.376+0000 7fd84b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.376+0000 7fd84b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.376+0000 7fd84b7fe640 1 -- 172.21.15.151:0/3321479830 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8540aa4b0 con 0x7fd8540b0ed0 2023-11-02T08:36:32.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.376+0000 7fc49ed9d640 1 --2- 172.21.15.151:0/613672604 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a0141c90 0x7fc4a013add0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc498009e50 tx=0x7fc498030260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a4d82b5044b266c server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 1 -- 172.21.15.151:0/2448039729 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8005ce0 con 0x7f4ed4038f10 2023-11-02T08:36:32.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 10 monclient: dump: 2023-11-02T08:36:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.605 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.605 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.605 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 1 -- 172.21.15.151:0/2448039729 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec8005e40 con 0x7f4ed4038f10 2023-11-02T08:36:32.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ee15d8640 5 monclient: authenticate success, global_id 4345 2023-11-02T08:36:32.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 1 -- 172.21.15.151:0/2448039729 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec80072f0 con 0x7f4ed4038f10 2023-11-02T08:36:32.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:32.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ed8ff9640 10 monclient: dump: 2023-11-02T08:36:32.607 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.607 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.607 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.608 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.608 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.608 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.608 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.608 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.609 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.609 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ee15d8640 10 monclient: _renew_subs 2023-11-02T08:36:32.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ee15d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed40c37f0 con 0x7f4ed4038f10 2023-11-02T08:36:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4eda7fc640 4 set_mon_vals no callback set 2023-11-02T08:36:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ee15d8640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ee15d8640 10 monclient: _renew_subs 2023-11-02T08:36:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ee15d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed40c0680 con 0x7f4ed4038f10 2023-11-02T08:36:32.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.377+0000 7f4ee15d8640 1 librados: init done 2023-11-02T08:36:32.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.378+0000 7f4ee15d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.378+0000 7f4ee15d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.378+0000 7f4ee15d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.611 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.378+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed4160030 con 0x7f4ed4038f10 2023-11-02T08:36:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5362ffd640 1 -- 172.21.15.151:0/2425773961 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c00cb40 con 0x7f537413e3b0 2023-11-02T08:36:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5362ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5362ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5362ffd640 10 monclient: dump: 2023-11-02T08:36:32.612 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.613 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.614 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.614 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.614 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.614 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.614 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5362ffd640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5362ffd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5362ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb717fff640 1 -- 172.21.15.151:0/2787737699 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0b6610 con 0x7fb724036360 2023-11-02T08:36:32.615 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb717fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb717fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:32.616 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb717fff640 10 monclient: dump: 2023-11-02T08:36:32.616 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.616 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.616 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.616 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.618 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb717fff640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb717fff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.618 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb717fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb731ff6640 5 monclient: authenticate success, global_id 4274 2023-11-02T08:36:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb731ff6640 10 monclient: _renew_subs 2023-11-02T08:36:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb731ff6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5362ffd640 1 -- 172.21.15.151:0/2425773961 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c00ccc0 con 0x7f537413e3b0 2023-11-02T08:36:32.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5362ffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f537359e640 5 monclient: authenticate success, global_id 4277 2023-11-02T08:36:32.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f537359e640 10 monclient: _renew_subs 2023-11-02T08:36:32.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f537359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53741409f0 con 0x7f537413e3b0 2023-11-02T08:36:32.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f5370d99640 4 set_mon_vals no callback set 2023-11-02T08:36:32.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7240c4f40 con 0x7fb724036360 2023-11-02T08:36:32.621 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f537359e640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f537359e640 10 monclient: _renew_subs 2023-11-02T08:36:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb731ff6640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb731ff6640 10 monclient: _renew_subs 2023-11-02T08:36:32.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb731ff6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7240a5b80 con 0x7fb724036360 2023-11-02T08:36:32.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7fb731ff6640 1 librados: init done 2023-11-02T08:36:32.623 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f537359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537413eb40 con 0x7f537413e3b0 2023-11-02T08:36:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.591+0000 7f537359e640 1 librados: init done 2023-11-02T08:36:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648802c650 con 0x7f647c0047c0 2023-11-02T08:36:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7f64927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7f64927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7f64927fc640 10 monclient: dump: 2023-11-02T08:36:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.626 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.627 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.627 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7f6498e1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:32.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7fb731ff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7fb731ff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7f537359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7f537359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7f537359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7fb731ff6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:32.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7240087b0 con 0x7fb724036360 2023-11-02T08:36:32.629 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.592+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5374140070 con 0x7f537413e3b0 2023-11-02T08:36:32.630 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.615+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648802c7b0 con 0x7f647c0047c0 2023-11-02T08:36:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.615+0000 7f64927fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.615+0000 7f64927fc640 1 -- 172.21.15.151:0/3682822413 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648802cb60 con 0x7f647c0047c0 2023-11-02T08:36:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.615+0000 7f64927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.615+0000 7f64927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:32.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.615+0000 7f64927fc640 10 monclient: dump: 2023-11-02T08:36:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.633 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.633 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.615+0000 7f6498e1b640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.615+0000 7f6498e1b640 4 set_mon_vals no callback set 2023-11-02T08:36:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.616+0000 7f6498e1b640 10 monclient: shutdown 2023-11-02T08:36:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.616+0000 7f6498e1b640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.616+0000 7f6498e1b640 1 -- 172.21.15.151:0/3682822413 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 msgr2=0x7f647c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.616+0000 7f6498e1b640 1 --2- 172.21.15.151:0/3682822413 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 0x7f647c004c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6488009a70 tx=0x7f647c0057f0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.616+0000 7f6498e1b640 1 -- 172.21.15.151:0/3682822413 shutdown_connections 2023-11-02T08:36:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.616+0000 7f6498e1b640 1 --2- 172.21.15.151:0/3682822413 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 0x7f647c004c40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.616+0000 7f6498e1b640 1 -- 172.21.15.151:0/3682822413 >> 172.21.15.151:0/3682822413 conn(0x7f64940b1850 msgr2=0x7f64940b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.616+0000 7f6498e1b640 1 -- 172.21.15.151:0/3682822413 shutdown_connections 2023-11-02T08:36:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.616+0000 7f6498e1b640 1 -- 172.21.15.151:0/3682822413 wait complete. 2023-11-02T08:36:32.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.617+0000 7f5362ffd640 1 -- 172.21.15.151:0/2425773961 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c01e640 con 0x7f537413e3b0 2023-11-02T08:36:32.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.617+0000 7f5362ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.617+0000 7f5362ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:32.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.617+0000 7f5362ffd640 10 monclient: dump: 2023-11-02T08:36:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.637 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.638 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.638 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.617+0000 7f6498e1b640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.640 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.617+0000 7f6498e1b640 10 monclient: monmap: 2023-11-02T08:36:32.640 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.640 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:32.640 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:32.640 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:32.641 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.641 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.641 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:32.641 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:32.641 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:32.642 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.617+0000 7f6498e1b640 1 librados: starting msgr at 2023-11-02T08:36:32.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.617+0000 7f6498e1b640 1 librados: starting objecter 2023-11-02T08:36:32.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.617+0000 7f6498e1b640 1 Processor -- start 2023-11-02T08:36:32.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 1 -- start start 2023-11-02T08:36:32.642 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 1 librados: setting wanted keys 2023-11-02T08:36:32.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 1 librados: calling monclient init 2023-11-02T08:36:32.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient: init 2023-11-02T08:36:32.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-02T08:36:32.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 0x7f64940a8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7fb717fff640 1 -- 172.21.15.151:0/2787737699 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c0b6770 con 0x7fb724036360 2023-11-02T08:36:32.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7fb717fff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7fb717fff640 1 -- 172.21.15.151:0/2787737699 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0b6ae0 con 0x7fb724036360 2023-11-02T08:36:32.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7fb717fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient(hunting): picked mon.noname-c con 0x7f647c0047c0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64940b5690 0x7f64940b2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.645 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient(hunting): picked mon.noname-a con 0x7f64940b5690 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:32.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7fb717fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:32.646 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7fb717fff640 10 monclient: dump: 2023-11-02T08:36:32.646 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.646 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.646 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.647 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.647 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.647 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.647 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.647 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.648 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.648 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7fb729ffb640 4 set_mon_vals no callback set 2023-11-02T08:36:32.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64941d4a80 0x7f6494135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient(hunting): picked mon.noname-b con 0x7f64941d4a80 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:32.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64940b4620 con 0x7f64940b5690 2023-11-02T08:36:32.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64941d4280 con 0x7f64941d4a80 2023-11-02T08:36:32.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64941d4560 con 0x7f647c0047c0 2023-11-02T08:36:32.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.618+0000 7f6498e1b640 10 monclient(hunting): authenticate will time out at 1363.974563s 2023-11-02T08:36:32.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.619+0000 7f6492ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64940b5690 0x7f64940b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.620+0000 7f6492ffd640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64940b5690 0x7f64940b2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38672/0 (socket says 172.21.15.151:38672) 2023-11-02T08:36:32.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.620+0000 7f6492ffd640 1 -- 172.21.15.151:0/3276966741 learned_addr learned my addr 172.21.15.151:0/3276966741 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.620+0000 7f6492ffd640 10 monclient(hunting): get_auth_request con 0x7f64940b5690 auth_method 0 2023-11-02T08:36:32.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.620+0000 7f6492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.620+0000 7f6492ffd640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.620+0000 7f6492ffd640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 1 -- 172.21.15.151:0/613672604 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498030e00 con 0x7fc4a0141c90 2023-11-02T08:36:32.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:32.653 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 10 monclient: dump: 2023-11-02T08:36:32.653 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.654 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.654 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.654 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.654 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.654 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.655 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.655 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.655 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.655 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.655 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 1 -- 172.21.15.151:0/613672604 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4980383f0 con 0x7fc4a0141c90 2023-11-02T08:36:32.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 5 monclient: authenticate success, global_id 4268 2023-11-02T08:36:32.656 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:32.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:32.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 1 -- 172.21.15.151:0/613672604 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498038780 con 0x7fc4a0141c90 2023-11-02T08:36:32.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 4 set_mon_vals no callback set 2023-11-02T08:36:32.657 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.658 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:32.658 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc49d59a640 10 monclient: dump: 2023-11-02T08:36:32.658 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.658 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.658 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.659 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.659 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.659 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.659 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.659 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.659 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.660 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7f6492ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7f6492ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7f6492ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 10 monclient: shutdown 2023-11-02T08:36:32.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 1 -- 172.21.15.151:0/613672604 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a0141c90 msgr2=0x7fc4a013add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 1 --2- 172.21.15.151:0/613672604 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a0141c90 0x7fc4a013add0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc498009e50 tx=0x7fc498030260 comp rx=0 tx=0).stop 2023-11-02T08:36:32.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 1 -- 172.21.15.151:0/613672604 shutdown_connections 2023-11-02T08:36:32.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 1 --2- 172.21.15.151:0/613672604 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a0141c90 0x7fc4a013add0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 1 --2- 172.21.15.151:0/613672604 >> v2:172.21.15.178:3300/0 conn(0x7fc4a006d830 0x7fc4a0141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.621+0000 7fc4a49f0640 1 -- 172.21.15.151:0/613672604 >> 172.21.15.151:0/613672604 conn(0x7fc4a006a520 msgr2=0x7fc4a00b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.622+0000 7fc4a49f0640 1 -- 172.21.15.151:0/613672604 shutdown_connections 2023-11-02T08:36:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.622+0000 7fc4a49f0640 1 -- 172.21.15.151:0/613672604 wait complete. 2023-11-02T08:36:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.622+0000 7fc4a49f0640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.622+0000 7fc4a49f0640 10 monclient: monmap: 2023-11-02T08:36:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.669 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:32.669 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:32.669 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:32.669 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.669 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.670 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:32.670 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:32.670 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:32.670 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 librados: starting msgr at 2023-11-02T08:36:32.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 librados: starting objecter 2023-11-02T08:36:32.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 Processor -- start 2023-11-02T08:36:32.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 -- start start 2023-11-02T08:36:32.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 librados: setting wanted keys 2023-11-02T08:36:32.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 librados: calling monclient init 2023-11-02T08:36:32.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient: init 2023-11-02T08:36:32.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-02T08:36:32.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4a006d830 0x7fc4a00a9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a006d830 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:32.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a00706b0 0x7fc4a00a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a00706b0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4a0141c90 0x7fc4a01d55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a0141c90 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:32.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.673 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00ab2c0 con 0x7fc4a0141c90 2023-11-02T08:36:32.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00ab530 con 0x7fc4a006d830 2023-11-02T08:36:32.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00a9f30 con 0x7fc4a00706b0 2023-11-02T08:36:32.674 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.623+0000 7fc4a49f0640 10 monclient(hunting): authenticate will time out at 1363.979701s 2023-11-02T08:36:32.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2023-11-02T08:36:32.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 1 -- 172.21.15.151:0/3276966741 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 msgr2=0x7f64940a8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 0x7f64940a8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 1 -- 172.21.15.151:0/3276966741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64941d4a80 msgr2=0x7f6494135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64941d4a80 0x7f6494135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.676 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 1 monclient: found mon.noname-a 2023-11-02T08:36:32.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:32.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.624+0000 7f6492ffd640 1 -- 172.21.15.151:0/3276966741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64940a9200 con 0x7f64940b5690 2023-11-02T08:36:32.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7fc49ed9d640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4a0141c90 0x7fc4a01d55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7f6492ffd640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64940b5690 0x7f64940b2940 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f648802c1f0 tx=0x7f6488004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7f647bfff640 10 client.?.objecter ms_handle_connect 0x7f64940b5690 2023-11-02T08:36:32.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7f647bfff640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:32.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7fc49ed9d640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4a0141c90 0x7fc4a01d55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38686/0 (socket says 172.21.15.151:38686) 2023-11-02T08:36:32.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7fc49ed9d640 1 -- 172.21.15.151:0/4106159933 learned_addr learned my addr 172.21.15.151:0/4106159933 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7fc49ed9d640 10 monclient(hunting): get_auth_request con 0x7fc4a0141c90 auth_method 0 2023-11-02T08:36:32.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7fc49ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7fc49ed9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.627+0000 7fc49ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.628+0000 7fc49ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.628+0000 7fc49ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.628+0000 7fc49ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2023-11-02T08:36:32.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 1 -- 172.21.15.151:0/4106159933 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a00706b0 msgr2=0x7fc4a00a96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a00706b0 0x7fc4a00a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 1 -- 172.21.15.151:0/4106159933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4a006d830 msgr2=0x7fc4a00a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4a006d830 0x7fc4a00a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 1 monclient: found mon.noname-a 2023-11-02T08:36:32.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:32.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.631+0000 7fc49ed9d640 1 -- 172.21.15.151:0/4106159933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc498009b30 con 0x7fc4a0141c90 2023-11-02T08:36:32.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.634+0000 7f2f33fff640 1 -- 172.21.15.151:0/54319016 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2f3c010c80 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.634+0000 7f2f33fff640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f2f3806be10 0x7f2f3806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.635+0000 7f2f4affd640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f2f3806be10 0x7f2f3806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.635+0000 7f2f4affd640 10 monclient: get_auth_request con 0x7f2f3806be10 auth_method 0 2023-11-02T08:36:32.684 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.635+0000 7f2f33fff640 10 client.4348.objecter ms_handle_connect 0x7f2f3806be10 2023-11-02T08:36:32.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.635+0000 7f2f4affd640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f2f3806be10 0x7f2f3806e2d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009870 tx=0x7f2f40030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.638+0000 7f647bfff640 1 -- 172.21.15.151:0/3276966741 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488034a90 con 0x7f64940b5690 2023-11-02T08:36:32.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.638+0000 7f647bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.638+0000 7f647bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:32.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.638+0000 7f647bfff640 10 monclient: dump: 2023-11-02T08:36:32.686 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.686 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.687 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.687 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.687 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.687 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.687 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.687 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.688 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.688 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.638+0000 7f647bfff640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.638+0000 7f647bfff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f647bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f647bfff640 1 -- 172.21.15.151:0/3276966741 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6488034bf0 con 0x7f64940b5690 2023-11-02T08:36:32.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6498e1b640 5 monclient: authenticate success, global_id 4305 2023-11-02T08:36:32.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6498e1b640 10 monclient: _renew_subs 2023-11-02T08:36:32.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6498e1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:32.689 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64941d4f10 con 0x7f64940b5690 2023-11-02T08:36:32.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f647bfff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7fc49ed9d640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4a0141c90 0x7fc4a01d55f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc49803a040 tx=0x7fc498004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7fc48effd640 10 client.?.objecter ms_handle_connect 0x7fc4a0141c90 2023-11-02T08:36:32.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7fc48effd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:32.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f647bfff640 1 -- 172.21.15.151:0/3276966741 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648803e410 con 0x7f64940b5690 2023-11-02T08:36:32.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f647bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f647bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:32.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f647bfff640 10 monclient: dump: 2023-11-02T08:36:32.692 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.692 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.692 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.692 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.693 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.693 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.693 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.693 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.693 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.694 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6498e1b640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:32.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6498e1b640 10 monclient: _renew_subs 2023-11-02T08:36:32.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6498e1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:32.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6494135820 con 0x7f64940b5690 2023-11-02T08:36:32.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6491ffb640 4 set_mon_vals no callback set 2023-11-02T08:36:32.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.639+0000 7f6498e1b640 1 librados: init done 2023-11-02T08:36:32.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.640+0000 7f6498e1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.640+0000 7f6498e1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.640+0000 7f6498e1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:32.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.640+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64941d4720 con 0x7f64940b5690 2023-11-02T08:36:32.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.870+0000 7faff17fa640 1 -- 172.21.15.151:0/3745728382 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fafe801a400 con 0x7fafec02d5e0 2023-11-02T08:36:32.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.870+0000 7faff17fa640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fafd806bdc0 0x7fafd806e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.870+0000 7faff8919640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fafd806bdc0 0x7fafd806e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.870+0000 7faff8919640 10 monclient: get_auth_request con 0x7fafd806bdc0 auth_method 0 2023-11-02T08:36:32.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.871+0000 7faff17fa640 10 client.4271.objecter ms_handle_connect 0x7fafd806bdc0 2023-11-02T08:36:32.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.871+0000 7faff8919640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fafd806bdc0 0x7fafd806e280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fafec008560 tx=0x7faff40a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.885 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.881+0000 7f4ed8ff9640 1 -- 172.21.15.151:0/2448039729 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4ec8007510 con 0x7f4ed4038f10 2023-11-02T08:36:32.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.882+0000 7f4ed8ff9640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f4ec006be10 0x7f4ec006e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.882+0000 7f4edbfff640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f4ec006be10 0x7f4ec006e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.882+0000 7f4edbfff640 10 monclient: get_auth_request con 0x7f4ec006be10 auth_method 0 2023-11-02T08:36:32.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.882+0000 7f4edbfff640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f4ec006be10 0x7f4ec006e2d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0000f60 tx=0x7f4ed0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.882+0000 7f4ed8ff9640 10 client.4345.objecter ms_handle_connect 0x7f4ec006be10 2023-11-02T08:36:32.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7f2f33fff640 1 -- 172.21.15.151:0/54319016 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2f3c0581c0 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7f2f33fff640 10 client.4348.objecter ms_dispatch 0x7f2f4c000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:32.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7f2f33fff640 3 client.4348.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:32.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7f2f33fff640 3 client.4348.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:32.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7f2f33fff640 20 client.4348.objecter dump_active .. 0 homeless 2023-11-02T08:36:32.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7faff17fa640 1 -- 172.21.15.151:0/3745728382 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fafe800f2e0 con 0x7fafec02d5e0 2023-11-02T08:36:32.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7faff17fa640 10 client.4271.objecter ms_dispatch 0x7faff4000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:32.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7faff17fa640 3 client.4271.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:32.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7faff17fa640 3 client.4271.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:32.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7faff17fa640 20 client.4271.objecter dump_active .. 0 homeless 2023-11-02T08:36:32.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7faff17fa640 1 -- 172.21.15.151:0/3745728382 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fafe80544b0 con 0x7fafec02d5e0 2023-11-02T08:36:32.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7faff17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.884+0000 7faff17fa640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:32.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 1 -- 172.21.15.151:0/4106159933 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498009510 con 0x7fc4a0141c90 2023-11-02T08:36:32.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:32.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 10 monclient: dump: 2023-11-02T08:36:32.894 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.894 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.894 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.895 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.895 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.895 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.895 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.896 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.896 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.896 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 1 -- 172.21.15.151:0/4106159933 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc498038e20 con 0x7fc4a0141c90 2023-11-02T08:36:32.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc4a49f0640 5 monclient: authenticate success, global_id 4308 2023-11-02T08:36:32.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 1 -- 172.21.15.151:0/4106159933 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498007bd0 con 0x7fc4a0141c90 2023-11-02T08:36:32.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:32.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc48effd640 10 monclient: dump: 2023-11-02T08:36:32.898 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.899 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.899 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.901 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.901 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.901 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.902 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.902 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.902 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.903 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc49cd99640 4 set_mon_vals no callback set 2023-11-02T08:36:32.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc4a49f0640 10 monclient: _renew_subs 2023-11-02T08:36:32.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc4a49f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:32.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a00a9c10 con 0x7fc4a0141c90 2023-11-02T08:36:32.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc4a49f0640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:32.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc4a49f0640 10 monclient: _renew_subs 2023-11-02T08:36:32.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc4a49f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:32.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a01d5d10 con 0x7fc4a0141c90 2023-11-02T08:36:32.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.886+0000 7fc4a49f0640 1 librados: init done 2023-11-02T08:36:32.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.887+0000 7f4ed8ff9640 1 -- 172.21.15.151:0/2448039729 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4ec8057f00 con 0x7f4ed4038f10 2023-11-02T08:36:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.887+0000 7f4ed8ff9640 10 client.4345.objecter ms_dispatch 0x7f4edc000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.887+0000 7f4ed8ff9640 3 client.4345.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.887+0000 7f4ed8ff9640 3 client.4345.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:32.909 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.887+0000 7f4ed8ff9640 20 client.4345.objecter dump_active .. 0 homeless 2023-11-02T08:36:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.887+0000 7fc4a49f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.887+0000 7fc4a49f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.887+0000 7fc4a49f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:32.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.887+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a0067920 con 0x7fc4a0141c90 2023-11-02T08:36:32.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.897+0000 7f5362ffd640 1 -- 172.21.15.151:0/2425773961 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f536c01d3b0 con 0x7f537413e3b0 2023-11-02T08:36:32.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.898+0000 7f5362ffd640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f535406be10 0x7f535406e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.898+0000 7f537259c640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f535406be10 0x7f535406e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.898+0000 7f537259c640 10 monclient: get_auth_request con 0x7f535406be10 auth_method 0 2023-11-02T08:36:32.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.898+0000 7f537259c640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f535406be10 0x7f535406e2d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5368001560 tx=0x7f5368007990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.898+0000 7f5362ffd640 10 client.4277.objecter ms_handle_connect 0x7f535406be10 2023-11-02T08:36:32.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.900+0000 7f647bfff640 1 -- 172.21.15.151:0/3276966741 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f648803b050 con 0x7f64940b5690 2023-11-02T08:36:32.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.900+0000 7f647bfff640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f648006bdf0 0x7f648006e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.916 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.900+0000 7f64937fe640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f648006bdf0 0x7f648006e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.900+0000 7f64937fe640 10 monclient: get_auth_request con 0x7f648006bdf0 auth_method 0 2023-11-02T08:36:32.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.901+0000 7f647bfff640 10 client.4305.objecter ms_handle_connect 0x7f648006bdf0 2023-11-02T08:36:32.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.901+0000 7f64937fe640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f648006bdf0 0x7f648006e2b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6484000fd0 tx=0x7f6484001590 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.907+0000 7fd84b7fe640 1 --2- 172.21.15.151:0/3321479830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b0ed0 0x7fd854144d40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0097c0 tx=0x7fd83c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2e391ad89707f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.907+0000 7f4ed8ff9640 1 -- 172.21.15.151:0/2448039729 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f4ec8053b40 con 0x7f4ed4038f10 2023-11-02T08:36:32.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.907+0000 7f4ed8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.907+0000 7f4ed8ff9640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:32.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.907+0000 7f2f33fff640 1 -- 172.21.15.151:0/54319016 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f2f3c053e00 con 0x7f2f4c0b75f0 2023-11-02T08:36:32.922 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.908+0000 7f2f33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.922 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.908+0000 7f2f33fff640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:32.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.915+0000 7f647bfff640 1 -- 172.21.15.151:0/3276966741 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6488079ad0 con 0x7f64940b5690 2023-11-02T08:36:32.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.915+0000 7f647bfff640 10 client.4305.objecter ms_dispatch 0x7f6494000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:32.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.915+0000 7f647bfff640 3 client.4305.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:32.924 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.915+0000 7f647bfff640 3 client.4305.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.915+0000 7f647bfff640 20 client.4305.objecter dump_active .. 0 homeless 2023-11-02T08:36:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.915+0000 7f647bfff640 1 -- 172.21.15.151:0/3276966741 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f6488075e10 con 0x7f64940b5690 2023-11-02T08:36:32.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.915+0000 7f647bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.915+0000 7f647bfff640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:32.937 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 1 -- 172.21.15.151:0/3321479830 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c031770 con 0x7fd8540b0ed0 2023-11-02T08:36:32.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:32.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 10 monclient: dump: 2023-11-02T08:36:32.939 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.939 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.939 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.940 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.940 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.940 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.941 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.941 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.941 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.942 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.943 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 1 -- 172.21.15.151:0/3321479830 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c0318d0 con 0x7fd8540b0ed0 2023-11-02T08:36:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 1 -- 172.21.15.151:0/3321479830 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c031c60 con 0x7fd8540b0ed0 2023-11-02T08:36:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:32.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd8537fe640 10 monclient: dump: 2023-11-02T08:36:32.945 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.946 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.946 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.946 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.947 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.947 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.947 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.948 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.948 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.948 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.933+0000 7fd85a527640 5 monclient: authenticate success, global_id 4354 2023-11-02T08:36:32.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.934+0000 7fd85a527640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:32.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.934+0000 7fd85a527640 4 set_mon_vals no callback set 2023-11-02T08:36:32.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.934+0000 7fd85a527640 10 monclient: shutdown 2023-11-02T08:36:32.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.935+0000 7fd85a527640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:32.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.935+0000 7fd85a527640 1 -- 172.21.15.151:0/3321479830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b0ed0 msgr2=0x7fd854144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.935+0000 7fd85a527640 1 --2- 172.21.15.151:0/3321479830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b0ed0 0x7fd854144d40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0097c0 tx=0x7fd83c02b810 comp rx=0 tx=0).stop 2023-11-02T08:36:32.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.935+0000 7fd85a527640 1 -- 172.21.15.151:0/3321479830 shutdown_connections 2023-11-02T08:36:32.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.935+0000 7fd85a527640 1 --2- 172.21.15.151:0/3321479830 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b0ed0 0x7fd854144d40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.935+0000 7fd85a527640 1 -- 172.21.15.151:0/3321479830 >> 172.21.15.151:0/3321479830 conn(0x7fd854065d50 msgr2=0x7fd854066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:32.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.935+0000 7fd85a527640 1 -- 172.21.15.151:0/3321479830 shutdown_connections 2023-11-02T08:36:32.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.935+0000 7fd85a527640 1 -- 172.21.15.151:0/3321479830 wait complete. 2023-11-02T08:36:32.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.936+0000 7fd85a527640 10 monclient: build_initial_monmap 2023-11-02T08:36:32.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.936+0000 7fd85a527640 10 monclient: monmap: 2023-11-02T08:36:32.956 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:32.956 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:32.956 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:32.956 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:32.956 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:32.957 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.957 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:32.957 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:32.957 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:32.957 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.937+0000 7fd85a527640 1 librados: starting msgr at 2023-11-02T08:36:32.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.937+0000 7fd85a527640 1 librados: starting objecter 2023-11-02T08:36:32.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.937+0000 7fd85a527640 1 Processor -- start 2023-11-02T08:36:32.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.937+0000 7fd85a527640 1 -- start start 2023-11-02T08:36:32.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.937+0000 7fd85a527640 1 librados: setting wanted keys 2023-11-02T08:36:32.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.937+0000 7fd85a527640 1 librados: calling monclient init 2023-11-02T08:36:32.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.937+0000 7fd85a527640 10 monclient: init 2023-11-02T08:36:32.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.938+0000 7fd85a527640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:32.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.938+0000 7fd85a527640 10 monclient: _add_conns ranks=[2,1,0] 2023-11-02T08:36:32.959 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.938+0000 7fd85a527640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd8540b0ed0 0x7fd854142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.938+0000 7fd85a527640 10 monclient(hunting): picked mon.noname-c con 0x7fd8540b0ed0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:32.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b4570 0x7fd8540ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 10 monclient(hunting): picked mon.noname-b con 0x7fd8540b4570 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:32.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd85413e520 0x7fd85413a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 10 monclient(hunting): picked mon.noname-a con 0x7fd85413e520 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:32.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8540adfa0 con 0x7fd85413e520 2023-11-02T08:36:32.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8540ac230 con 0x7fd8540b4570 2023-11-02T08:36:32.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:32.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8540ada20 con 0x7fd8540b0ed0 2023-11-02T08:36:32.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:32.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd85a527640 10 monclient(hunting): authenticate will time out at 1364.295294s 2023-11-02T08:36:32.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd84b7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b4570 0x7fd8540ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd84b7fe640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b4570 0x7fd8540ad4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36372/0 (socket says 172.21.15.151:36372) 2023-11-02T08:36:32.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd84b7fe640 1 -- 172.21.15.151:0/3865031720 learned_addr learned my addr 172.21.15.151:0/3865031720 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:32.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd84b7fe640 10 monclient(hunting): get_auth_request con 0x7fd8540b4570 auth_method 0 2023-11-02T08:36:32.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd84b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:32.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd84b7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:32.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.939+0000 7fd84b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:32.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.940+0000 7fd84b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:32.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.940+0000 7fd84b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:32.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.940+0000 7fd84b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:32.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2023-11-02T08:36:32.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 1 -- 172.21.15.151:0/3865031720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd8540b0ed0 msgr2=0x7fd854142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd8540b0ed0 0x7fd854142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 1 -- 172.21.15.151:0/3865031720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd85413e520 msgr2=0x7fd85413a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:32.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd85413e520 0x7fd85413a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:32.965 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 10 monclient: _finish_hunting 0 2023-11-02T08:36:32.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 1 monclient: found mon.noname-b 2023-11-02T08:36:32.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:32.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.943+0000 7fd84b7fe640 1 -- 172.21.15.151:0/3865031720 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c0093a0 con 0x7fd8540b4570 2023-11-02T08:36:32.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.947+0000 7fd84b7fe640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b4570 0x7fd8540ad4e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd83c034040 tx=0x7fd83c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.947+0000 7fd8517fa640 10 client.?.objecter ms_handle_connect 0x7fd8540b4570 2023-11-02T08:36:32.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.947+0000 7fd8517fa640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:32.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 1 -- 172.21.15.151:0/3865031720 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c032990 con 0x7fd8540b4570 2023-11-02T08:36:32.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:32.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 10 monclient: dump: 2023-11-02T08:36:32.971 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.971 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.971 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.971 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.972 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.972 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.972 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.972 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.972 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.973 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 10 monclient: _finish_auth 0 2023-11-02T08:36:32.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 10 monclient: _check_auth_tickets 2023-11-02T08:36:32.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:32.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 1 -- 172.21.15.151:0/3865031720 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c032b10 con 0x7fd8540b4570 2023-11-02T08:36:32.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:32.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 1 -- 172.21.15.151:0/3865031720 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0453f0 con 0x7fd8540b4570 2023-11-02T08:36:32.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:32.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:32.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd8517fa640 10 monclient: dump: 2023-11-02T08:36:32.975 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:32.975 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:32.975 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.975 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:32.975 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:32.976 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:32.976 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:32.976 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:32.976 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:32.976 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:32.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.951+0000 7fd852ffd640 4 set_mon_vals no callback set 2023-11-02T08:36:32.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.952+0000 7fd85a527640 5 monclient: authenticate success, global_id 4360 2023-11-02T08:36:32.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.952+0000 7fd85a527640 10 monclient: _renew_subs 2023-11-02T08:36:32.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.952+0000 7fd85a527640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.977 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.952+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8540adc40 con 0x7fd8540b4570 2023-11-02T08:36:32.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.952+0000 7fd85a527640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:32.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.952+0000 7fd85a527640 10 monclient: _renew_subs 2023-11-02T08:36:32.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.952+0000 7fd85a527640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.952+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85413ac00 con 0x7fd8540b4570 2023-11-02T08:36:32.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.952+0000 7fd85a527640 1 librados: init done 2023-11-02T08:36:32.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.954+0000 7fd85a527640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.954+0000 7fd85a527640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:32.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.954+0000 7fd85a527640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:32.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.954+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8540ae110 con 0x7fd8540b4570 2023-11-02T08:36:32.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.970+0000 7fd8517fa640 1 -- 172.21.15.151:0/3865031720 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd83c03b3f0 con 0x7fd8540b4570 2023-11-02T08:36:32.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.971+0000 7fd8517fa640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fd83406bdc0 0x7fd83406e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:32.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.971+0000 7fd853fff640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fd83406bdc0 0x7fd83406e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:32.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.971+0000 7fd853fff640 10 monclient: get_auth_request con 0x7fd83406bdc0 auth_method 0 2023-11-02T08:36:32.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.971+0000 7fd853fff640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fd83406bdc0 0x7fd83406e280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd8440020a0 tx=0x7fd844001690 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:32.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:32.972+0000 7fd8517fa640 10 client.4360.objecter ms_handle_connect 0x7fd83406bdc0 2023-11-02T08:36:33.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.098+0000 7f6498e1b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-02T08:36:33.102 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.098+0000 7f6498e1b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-02T08:36:33.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.098+0000 7f6498e1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:33.103 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.098+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f649413bb20 con 0x7f64940b5690 2023-11-02T08:36:33.114 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.111+0000 7f5362ffd640 1 -- 172.21.15.151:0/2425773961 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f536c00ce30 con 0x7f537413e3b0 2023-11-02T08:36:33.115 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.111+0000 7f5362ffd640 10 client.4277.objecter ms_dispatch 0x7f5374000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:33.115 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.111+0000 7f5362ffd640 3 client.4277.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:33.116 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.111+0000 7f5362ffd640 3 client.4277.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:33.116 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.111+0000 7f5362ffd640 20 client.4277.objecter dump_active .. 0 homeless 2023-11-02T08:36:33.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.198+0000 7fd8517fa640 1 -- 172.21.15.151:0/3865031720 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd83c07aac0 con 0x7fd8540b4570 2023-11-02T08:36:33.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.198+0000 7fd8517fa640 10 client.4360.objecter ms_dispatch 0x7fd854000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:33.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.198+0000 7fd8517fa640 3 client.4360.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:33.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.198+0000 7fd8517fa640 3 client.4360.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:33.203 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.198+0000 7fd8517fa640 20 client.4360.objecter dump_active .. 0 homeless 2023-11-02T08:36:33.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.208+0000 7fd8517fa640 1 -- 172.21.15.151:0/3865031720 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fd83c076700 con 0x7fd8540b4570 2023-11-02T08:36:33.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.208+0000 7fd8517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:33.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.208+0000 7fd8517fa640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:33.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.209+0000 7f4ee15d8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-02T08:36:33.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.209+0000 7f4ee15d8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-02T08:36:33.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.210+0000 7f4ee15d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:33.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.210+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f4ed4030250 con 0x7f4ed4038f10 2023-11-02T08:36:33.217 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.214+0000 7f4ed8ff9640 1 -- 172.21.15.151:0/2448039729 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4ec8017090 con 0x7f4ed4038f10 2023-11-02T08:36:33.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.214+0000 7f4ed8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-02T08:36:33.218 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.214+0000 7f4ed8ff9640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:33.218 INFO:teuthology.orchestra.run.smithi151.stdout:60129542148 2023-11-02T08:36:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 10 librados: watch_flush enter 2023-11-02T08:36:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 10 librados: watch_flush exit 2023-11-02T08:36:33.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 20 client.4345.objecter shutdown clearing up homeless session... 2023-11-02T08:36:33.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 10 client.4345.objecter successfully canceled tick 2023-11-02T08:36:33.222 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f4ec006be10 msgr2=0x7f4ec006e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f4ec006be10 0x7f4ec006e2d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0000f60 tx=0x7f4ed0009040 comp rx=0 tx=0).stop 2023-11-02T08:36:33.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 10 monclient: shutdown 2023-11-02T08:36:33.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:33.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed4038f10 msgr2=0x7f4ed402fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed4038f10 0x7f4ed402fbd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800c410 tx=0x7f4ec800c8d0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 shutdown_connections 2023-11-02T08:36:33.226 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4ed40d31b0 0x7f4ed40c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f4ec006be10 0x7f4ec006e2d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed4038f10 0x7f4ed402fbd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.227 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 --2- 172.21.15.151:0/2448039729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4ed40357e0 0x7f4ed40c2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 >> 172.21.15.151:0/2448039729 conn(0x7f4ed4006690 msgr2=0x7f4ed40328a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:33.228 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.216+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 shutdown_connections 2023-11-02T08:36:33.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.217+0000 7f4ee15d8640 1 -- 172.21.15.151:0/2448039729 wait complete. 2023-11-02T08:36:33.229 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.217+0000 7f4ee15d8640 1 librados: shutdown 2023-11-02T08:36:33.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.221+0000 7fc48effd640 1 -- 172.21.15.151:0/4106159933 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc498007d30 con 0x7fc4a0141c90 2023-11-02T08:36:33.230 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.221+0000 7fc48effd640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc48006be10 0x7fc48006e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:33.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.222+0000 7fc49e59c640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc48006be10 0x7fc48006e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:33.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.222+0000 7fc49e59c640 10 monclient: get_auth_request con 0x7fc48006be10 auth_method 0 2023-11-02T08:36:33.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.222+0000 7fc49e59c640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc48006be10 0x7fc48006e2d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc490001560 tx=0x7fc490007990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:33.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.222+0000 7fc48effd640 10 client.4308.objecter ms_handle_connect 0x7fc48006be10 2023-11-02T08:36:33.232 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.6 2023-11-02T08:36:33.252 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.248+0000 7fc48effd640 1 -- 172.21.15.151:0/4106159933 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc49807fe40 con 0x7fc4a0141c90 2023-11-02T08:36:33.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.248+0000 7fc48effd640 10 client.4308.objecter ms_dispatch 0x7fc4a0000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:33.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.248+0000 7fc48effd640 3 client.4308.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:33.253 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.248+0000 7fc48effd640 3 client.4308.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:33.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.248+0000 7fc48effd640 20 client.4308.objecter dump_active .. 0 homeless 2023-11-02T08:36:33.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.248+0000 7fc48effd640 1 -- 172.21.15.151:0/4106159933 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fc49807ba80 con 0x7fc4a0141c90 2023-11-02T08:36:33.254 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.248+0000 7fc48effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:33.255 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.248+0000 7fc48effd640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:33.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.258+0000 7faffab7d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-02T08:36:33.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.258+0000 7faffab7d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-02T08:36:33.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.258+0000 7faffab7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:33.262 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.258+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fafec0ca3e0 con 0x7fafec02d5e0 2023-11-02T08:36:33.280 INFO:teuthology.orchestra.run.smithi151.stdout:55834574851 2023-11-02T08:36:33.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.276+0000 7f647bfff640 1 -- 172.21.15.151:0/3276966741 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6488039090 con 0x7f64940b5690 2023-11-02T08:36:33.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.276+0000 7f647bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-02T08:36:33.281 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.276+0000 7f647bfff640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:33.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 10 librados: watch_flush enter 2023-11-02T08:36:33.284 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 10 librados: watch_flush exit 2023-11-02T08:36:33.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 20 client.4305.objecter shutdown clearing up homeless session... 2023-11-02T08:36:33.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 10 client.4305.objecter successfully canceled tick 2023-11-02T08:36:33.285 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f648006bdf0 msgr2=0x7f648006e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f648006bdf0 0x7f648006e2b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6484000fd0 tx=0x7f6484001590 comp rx=0 tx=0).stop 2023-11-02T08:36:33.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 10 monclient: shutdown 2023-11-02T08:36:33.286 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:33.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64940b5690 msgr2=0x7f64940b2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.287 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64940b5690 0x7f64940b2940 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f648802c1f0 tx=0x7f6488004730 comp rx=0 tx=0).stop 2023-11-02T08:36:33.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 shutdown_connections 2023-11-02T08:36:33.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f648006bdf0 0x7f648006e2b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.288 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64941d4a80 0x7f6494135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.289 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64940b5690 0x7f64940b2940 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.289 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 --2- 172.21.15.151:0/3276966741 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f647c0047c0 0x7f64940a8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.289 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 >> 172.21.15.151:0/3276966741 conn(0x7f64940b1850 msgr2=0x7f64940afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:33.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 shutdown_connections 2023-11-02T08:36:33.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.278+0000 7f6498e1b640 1 -- 172.21.15.151:0/3276966741 wait complete. 2023-11-02T08:36:33.290 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.279+0000 7f6498e1b640 1 librados: shutdown 2023-11-02T08:36:33.292 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.285+0000 7f2f51841640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-02T08:36:33.292 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.285+0000 7f2f51841640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-02T08:36:33.293 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.285+0000 7f2f51841640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:33.293 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.285+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2f4c140760 con 0x7f2f4c0b75f0 2023-11-02T08:36:33.293 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.288+0000 7f2f33fff640 1 -- 172.21.15.151:0/54319016 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2f4c140760 con 0x7f2f4c0b75f0 2023-11-02T08:36:33.294 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.288+0000 7f2f33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-02T08:36:33.294 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.288+0000 7f2f33fff640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:33.294 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:33.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2023-11-02T08:36:33.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 10 librados: watch_flush enter 2023-11-02T08:36:33.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 10 librados: watch_flush exit 2023-11-02T08:36:33.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 20 client.4348.objecter shutdown clearing up homeless session... 2023-11-02T08:36:33.296 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 10 client.4348.objecter successfully canceled tick 2023-11-02T08:36:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f5362ffd640 1 -- 172.21.15.151:0/2425773961 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f536c0541b0 con 0x7f537413e3b0 2023-11-02T08:36:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f5362ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:33.297 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f5362ffd640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f2f3806be10 msgr2=0x7f2f3806e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f2f3806be10 0x7f2f3806e2d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009870 tx=0x7f2f40030000 comp rx=0 tx=0).stop 2023-11-02T08:36:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 10 monclient: shutdown 2023-11-02T08:36:33.299 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:33.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f4c0b75f0 msgr2=0x7f2f4c1d61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f4c0b75f0 0x7f2f4c1d61c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00b4a0 tx=0x7f2f3c00b960 comp rx=0 tx=0).stop 2023-11-02T08:36:33.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 shutdown_connections 2023-11-02T08:36:33.300 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f4c13ec50 0x7f2f4c1d6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f2f3806be10 0x7f2f3806e2d0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009870 tx=0x7f2f40030000 comp rx=0 tx=0).stop 2023-11-02T08:36:33.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f4c0b75f0 0x7f2f4c1d61c0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00b4a0 tx=0x7f2f3c00b960 comp rx=0 tx=0).stop 2023-11-02T08:36:33.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 --2- 172.21.15.151:0/54319016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f4c0a8c90 0x7f2f4c1cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.301 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.291+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 >> 172.21.15.151:0/54319016 conn(0x7f2f4c065d50 msgr2=0x7f2f4c0b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:33.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.292+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 shutdown_connections 2023-11-02T08:36:33.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.292+0000 7f2f51841640 1 -- 172.21.15.151:0/54319016 wait complete. 2023-11-02T08:36:33.302 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.292+0000 7f2f51841640 1 librados: shutdown 2023-11-02T08:36:33.308 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.2 2023-11-02T08:36:33.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.342+0000 7fb717fff640 1 -- 172.21.15.151:0/2787737699 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb72c0b6c40 con 0x7fb724036360 2023-11-02T08:36:33.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.342+0000 7fb717fff640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fb71006bdc0 0x7fb71006e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:33.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.342+0000 7fb72affd640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fb71006bdc0 0x7fb71006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:33.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.342+0000 7fb72affd640 10 monclient: get_auth_request con 0x7fb71006bdc0 auth_method 0 2023-11-02T08:36:33.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.343+0000 7fb72affd640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fb71006bdc0 0x7fb71006e280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb7240c53b0 tx=0x7fb720009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:33.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.343+0000 7fb717fff640 10 client.4274.objecter ms_handle_connect 0x7fb71006bdc0 2023-11-02T08:36:33.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.364+0000 7fb717fff640 1 -- 172.21.15.151:0/2787737699 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb72c0f46d0 con 0x7fb724036360 2023-11-02T08:36:33.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.364+0000 7fb717fff640 10 client.4274.objecter ms_dispatch 0x7fb72c000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:33.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.364+0000 7fb717fff640 3 client.4274.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:33.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.364+0000 7fb717fff640 3 client.4274.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:33.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.364+0000 7fb717fff640 20 client.4274.objecter dump_active .. 0 homeless 2023-11-02T08:36:33.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.364+0000 7fb717fff640 1 -- 172.21.15.151:0/2787737699 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fb72c0f0310 con 0x7fb724036360 2023-11-02T08:36:33.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.364+0000 7fb717fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:33.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.364+0000 7fb717fff640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:33.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.391+0000 7faff17fa640 1 -- 172.21.15.151:0/3745728382 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fafec0ca3e0 con 0x7fafec02d5e0 2023-11-02T08:36:33.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.391+0000 7faff17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-02T08:36:33.395 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.391+0000 7faff17fa640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:33.396 INFO:teuthology.orchestra.run.smithi151.stdout:60129542148 2023-11-02T08:36:33.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 10 librados: watch_flush enter 2023-11-02T08:36:33.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 10 librados: watch_flush exit 2023-11-02T08:36:33.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 20 client.4271.objecter shutdown clearing up homeless session... 2023-11-02T08:36:33.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 10 client.4271.objecter successfully canceled tick 2023-11-02T08:36:33.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fafd806bdc0 msgr2=0x7fafd806e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.400 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fafd806bdc0 0x7fafd806e280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fafec008560 tx=0x7faff40a7000 comp rx=0 tx=0).stop 2023-11-02T08:36:33.401 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 10 monclient: shutdown 2023-11-02T08:36:33.401 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:33.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec02d5e0 msgr2=0x7fafec0c5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec02d5e0 0x7fafec0c5920 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fafe800b490 tx=0x7fafe800b950 comp rx=0 tx=0).stop 2023-11-02T08:36:33.402 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 shutdown_connections 2023-11-02T08:36:33.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fafec02d5e0 0x7fafec0c5920 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.403 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fafd806bdc0 0x7fafd806e280 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafec030ae0 0x7fafec02ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 --2- 172.21.15.151:0/3745728382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fafec006fd0 0x7fafec0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.404 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 >> 172.21.15.151:0/3745728382 conn(0x7fafec0061f0 msgr2=0x7fafec03c950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:33.405 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 shutdown_connections 2023-11-02T08:36:33.405 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 -- 172.21.15.151:0/3745728382 wait complete. 2023-11-02T08:36:33.406 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.394+0000 7faffab7d640 1 librados: shutdown 2023-11-02T08:36:33.414 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.4 2023-11-02T08:36:33.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.464+0000 7fc4a49f0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-02T08:36:33.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.464+0000 7fc4a49f0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-02T08:36:33.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.464+0000 7fc4a49f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:33.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.464+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc4a00aa0f0 con 0x7fc4a0141c90 2023-11-02T08:36:33.519 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.516+0000 7fd85a527640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-02T08:36:33.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.516+0000 7fd85a527640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-02T08:36:33.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.516+0000 7fd85a527640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:33.520 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.516+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd854067920 con 0x7fd8540b4570 2023-11-02T08:36:33.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.520+0000 7fd8517fa640 1 -- 172.21.15.151:0/3865031720 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd83c079d20 con 0x7fd8540b4570 2023-11-02T08:36:33.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.520+0000 7fd8517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-02T08:36:33.524 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.520+0000 7fd8517fa640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:33.525 INFO:teuthology.orchestra.run.smithi151.stdout:60129542148 2023-11-02T08:36:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 10 librados: watch_flush enter 2023-11-02T08:36:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 10 librados: watch_flush exit 2023-11-02T08:36:33.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 20 client.4360.objecter shutdown clearing up homeless session... 2023-11-02T08:36:33.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 10 client.4360.objecter successfully canceled tick 2023-11-02T08:36:33.528 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fd83406bdc0 msgr2=0x7fd83406e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fd83406bdc0 0x7fd83406e280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd8440020a0 tx=0x7fd844001690 comp rx=0 tx=0).stop 2023-11-02T08:36:33.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 10 monclient: shutdown 2023-11-02T08:36:33.529 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:33.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b4570 msgr2=0x7fd8540ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b4570 0x7fd8540ad4e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd83c034040 tx=0x7fd83c0046c0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.530 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 shutdown_connections 2023-11-02T08:36:33.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd85413e520 0x7fd85413a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fd83406bdc0 0x7fd83406e280 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.531 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8540b4570 0x7fd8540ad4e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 --2- 172.21.15.151:0/3865031720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd8540b0ed0 0x7fd854142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.532 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 >> 172.21.15.151:0/3865031720 conn(0x7fd854065d50 msgr2=0x7fd85413ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:33.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 shutdown_connections 2023-11-02T08:36:33.533 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 -- 172.21.15.151:0/3865031720 wait complete. 2023-11-02T08:36:33.534 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.523+0000 7fd85a527640 1 librados: shutdown 2023-11-02T08:36:33.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.5 2023-11-02T08:36:33.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.560+0000 7fb731ff6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-02T08:36:33.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.561+0000 7fb731ff6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-02T08:36:33.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.561+0000 7fb731ff6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:33.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.561+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb7240a5d60 con 0x7fb724036360 2023-11-02T08:36:33.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.565+0000 7fb717fff640 1 -- 172.21.15.151:0/2787737699 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb72c0f3930 con 0x7fb724036360 2023-11-02T08:36:33.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.565+0000 7fb717fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-02T08:36:33.568 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.565+0000 7fb717fff640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:33.569 INFO:teuthology.orchestra.run.smithi151.stdout:60129542147 2023-11-02T08:36:33.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 10 librados: watch_flush enter 2023-11-02T08:36:33.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 10 librados: watch_flush exit 2023-11-02T08:36:33.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 20 client.4274.objecter shutdown clearing up homeless session... 2023-11-02T08:36:33.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 10 client.4274.objecter successfully canceled tick 2023-11-02T08:36:33.571 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fb71006bdc0 msgr2=0x7fb71006e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fb71006bdc0 0x7fb71006e280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb7240c53b0 tx=0x7fb720009060 comp rx=0 tx=0).stop 2023-11-02T08:36:33.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 10 monclient: shutdown 2023-11-02T08:36:33.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:33.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb724036360 msgr2=0x7fb7240a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.572 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb724036360 0x7fb7240a6390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0727b0 tx=0x7fb72c068a40 comp rx=0 tx=0).stop 2023-11-02T08:36:33.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 shutdown_connections 2023-11-02T08:36:33.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fb71006bdc0 0x7fb71006e280 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7240cbb50 0x7fb7240c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7240c53e0 0x7fb7240c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.573 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 --2- 172.21.15.151:0/2787737699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb724036360 0x7fb7240a6390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 >> 172.21.15.151:0/2787737699 conn(0x7fb724006720 msgr2=0x7fb7240cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:33.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 shutdown_connections 2023-11-02T08:36:33.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 -- 172.21.15.151:0/2787737699 wait complete. 2023-11-02T08:36:33.574 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.567+0000 7fb731ff6640 1 librados: shutdown 2023-11-02T08:36:33.581 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.3 2023-11-02T08:36:33.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.589+0000 7f537359e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-02T08:36:33.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.589+0000 7f537359e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-02T08:36:33.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.589+0000 7f537359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:33.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.589+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5374140be0 con 0x7f537413e3b0 2023-11-02T08:36:33.596 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.593+0000 7f5362ffd640 1 -- 172.21.15.151:0/2425773961 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f536c019090 con 0x7f537413e3b0 2023-11-02T08:36:33.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.593+0000 7f5362ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-02T08:36:33.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.593+0000 7f5362ffd640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:33.597 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:33.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 10 librados: watch_flush enter 2023-11-02T08:36:33.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 10 librados: watch_flush exit 2023-11-02T08:36:33.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 20 client.4277.objecter shutdown clearing up homeless session... 2023-11-02T08:36:33.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 10 client.4277.objecter successfully canceled tick 2023-11-02T08:36:33.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f535406be10 msgr2=0x7f535406e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f535406be10 0x7f535406e2d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5368001560 tx=0x7f5368007990 comp rx=0 tx=0).stop 2023-11-02T08:36:33.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 10 monclient: shutdown 2023-11-02T08:36:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:33.602 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f537413e3b0 msgr2=0x7f53741404b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f537413e3b0 0x7f53741404b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f536c00b970 tx=0x7f536c00be30 comp rx=0 tx=0).stop 2023-11-02T08:36:33.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 shutdown_connections 2023-11-02T08:36:33.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f537413e3b0 0x7f53741404b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.603 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f535406be10 0x7f535406e2d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53740add80 0x7f537406cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.595+0000 7f537359e640 1 --2- 172.21.15.151:0/2425773961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f53740a8720 0x7f537413a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.596+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 >> 172.21.15.151:0/2425773961 conn(0x7f53740b2ae0 msgr2=0x7f53740b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:33.604 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.596+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 shutdown_connections 2023-11-02T08:36:33.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.596+0000 7f537359e640 1 -- 172.21.15.151:0/2425773961 wait complete. 2023-11-02T08:36:33.605 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.596+0000 7f537359e640 1 librados: shutdown 2023-11-02T08:36:33.610 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.7 2023-11-02T08:36:33.627 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.624+0000 7fc48effd640 1 -- 172.21.15.151:0/4106159933 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc498047090 con 0x7fc4a0141c90 2023-11-02T08:36:33.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.624+0000 7fc48effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-02T08:36:33.628 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.624+0000 7fc48effd640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:33.628 INFO:teuthology.orchestra.run.smithi151.stdout:60129542148 2023-11-02T08:36:33.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 10 librados: watch_flush enter 2023-11-02T08:36:33.631 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 10 librados: watch_flush exit 2023-11-02T08:36:33.632 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 20 client.4308.objecter shutdown clearing up homeless session... 2023-11-02T08:36:33.632 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 10 client.4308.objecter successfully canceled tick 2023-11-02T08:36:33.633 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc48006be10 msgr2=0x7fc48006e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc48006be10 0x7fc48006e2d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc490001560 tx=0x7fc490007990 comp rx=0 tx=0).stop 2023-11-02T08:36:33.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 10 monclient: shutdown 2023-11-02T08:36:33.634 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:33.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4a0141c90 msgr2=0x7fc4a01d55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:33.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4a0141c90 0x7fc4a01d55f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc49803a040 tx=0x7fc498004a40 comp rx=0 tx=0).stop 2023-11-02T08:36:33.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 shutdown_connections 2023-11-02T08:36:33.635 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc4a0141c90 0x7fc4a01d55f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc4a00706b0 0x7fc4a00a96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc48006be10 0x7fc48006e2d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.636 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 --2- 172.21.15.151:0/4106159933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4a006d830 0x7fc4a00a9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:33.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 >> 172.21.15.151:0/4106159933 conn(0x7fc4a006a520 msgr2=0x7fc4a0070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:33.637 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 shutdown_connections 2023-11-02T08:36:33.638 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.626+0000 7fc4a49f0640 1 -- 172.21.15.151:0/4106159933 wait complete. 2023-11-02T08:36:33.639 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:33.627+0000 7fc4a49f0640 1 librados: shutdown 2023-11-02T08:36:33.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.1 2023-11-02T08:36:34.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-02T08:36:34.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-02T08:36:34.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-02T08:36:34.337 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.334+0000 7f85f2c1c640 10 monclient: get_monmap_and_config 2023-11-02T08:36:34.337 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.334+0000 7f85f2c1c640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.334+0000 7f85f2c1c640 10 monclient: monmap: 2023-11-02T08:36:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.338 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:34.335126+0000 2023-11-02T08:36:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:34.335126+0000 2023-11-02T08:36:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.339 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:34.340 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:34.340 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.342 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.334+0000 7f85f2c1c640 1 Processor -- start 2023-11-02T08:36:34.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.334+0000 7f85f2c1c640 1 -- start start 2023-11-02T08:36:34.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.334+0000 7f85f2c1c640 10 monclient: init 2023-11-02T08:36:34.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-11-02T08:36:34.343 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f85ec0b4570 0x7f85ec144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 10 monclient(hunting): picked mon.noname-b con 0x7f85ec0b4570 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f85ec0b0ed0 0x7f85ec0a9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 10 monclient(hunting): picked mon.noname-c con 0x7f85ec0b0ed0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:34.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85ec0b1cb0 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:34.344 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec0aa1d0 con 0x7f85ec0b1cb0 2023-11-02T08:36:34.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec0b4950 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec0aa4b0 con 0x7f85ec0b4570 2023-11-02T08:36:34.345 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.335+0000 7f85f2c1c640 10 monclient(hunting): authenticate will time out at 1365.691352s 2023-11-02T08:36:34.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85f11b9640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f85ec0b1cb0 0x7f85ec0b2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:34152/0 (socket says 172.21.15.151:34152) 2023-11-02T08:36:34.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85f11b9640 1 -- 172.21.15.151:0/3630004042 learned_addr learned my addr 172.21.15.151:0/3630004042 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:34.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85ebfff640 1 --2- 172.21.15.151:0/3630004042 >> v2:172.21.15.178:3300/0 conn(0x7f85ec0b0ed0 0x7f85ec0a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.346 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85f09b8640 1 --2- 172.21.15.151:0/3630004042 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f85ec0b4570 0x7f85ec144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85ebfff640 10 monclient(hunting): get_auth_request con 0x7f85ec0b0ed0 auth_method 0 2023-11-02T08:36:34.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85ebfff640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.347 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85ebfff640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.348 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85f09b8640 10 monclient(hunting): get_auth_request con 0x7f85ec0b4570 auth_method 0 2023-11-02T08:36:34.348 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85f09b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.348 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85f09b8640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.350 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.336+0000 7f85f09b8640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.350 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.337+0000 7f85ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.350 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.337+0000 7f85ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.350 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.337+0000 7f85ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85ebfff640 10 monclient(hunting): handle_auth_done global_id 4366 payload 274 2023-11-02T08:36:34.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85ebfff640 1 -- 172.21.15.151:0/3630004042 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f85ec0b4570 msgr2=0x7f85ec144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.355 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85ebfff640 1 --2- 172.21.15.151:0/3630004042 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f85ec0b4570 0x7f85ec144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85eb7fe640 1 -- 172.21.15.151:0/3630004042 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253747213 0 0) 0x7f85ec0aa1d0 con 0x7f85ec0b1cb0 2023-11-02T08:36:34.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85ebfff640 1 -- 172.21.15.151:0/3630004042 >> v1:172.21.15.151:6789/0 conn(0x7f85ec0b1cb0 legacy=0x7f85ec0b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.356 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85ebfff640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85ebfff640 1 monclient: found mon.noname-c 2023-11-02T08:36:34.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85ebfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.357 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85ebfff640 1 -- 172.21.15.151:0/3630004042 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec0a8b50 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.338+0000 7f85eb7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-11-02T08:36:34.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.340+0000 7f85ebfff640 1 --2- 172.21.15.151:0/3630004042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 0x7f85ec0a9070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009790 tx=0x7f85d802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=305252d0982d56a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.358 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 1 -- 172.21.15.151:0/3630004042 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d80316f0 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:34.359 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 10 monclient: dump: 2023-11-02T08:36:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.361 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.361 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.361 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.362 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.362 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.362 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.363 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.363 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.363 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.364 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 1 -- 172.21.15.151:0/3630004042 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d8031850 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 1 -- 172.21.15.151:0/3630004042 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d8031be0 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.365 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:34.366 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85eb7fe640 10 monclient: dump: 2023-11-02T08:36:34.366 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.367 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.367 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.367 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.367 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.367 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.368 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.368 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.368 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.368 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.368 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85f2c1c640 5 monclient: authenticate success, global_id 4366 2023-11-02T08:36:34.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85f2c1c640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:34.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.344+0000 7f85f2c1c640 4 set_mon_vals no callback set 2023-11-02T08:36:34.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 10 monclient: shutdown 2023-11-02T08:36:34.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:34.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 1 -- 172.21.15.151:0/3630004042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 msgr2=0x7f85ec0a9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.369 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 1 --2- 172.21.15.151:0/3630004042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 0x7f85ec0a9070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009790 tx=0x7f85d802b790 comp rx=0 tx=0).stop 2023-11-02T08:36:34.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 1 -- 172.21.15.151:0/3630004042 shutdown_connections 2023-11-02T08:36:34.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 1 --2- 172.21.15.151:0/3630004042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 0x7f85ec0a9070 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 1 --2- 172.21.15.151:0/3630004042 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f85ec0b4570 0x7f85ec144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 1 -- 172.21.15.151:0/3630004042 >> 172.21.15.151:0/3630004042 conn(0x7f85ec065d50 msgr2=0x7f85ec066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:34.370 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 1 -- 172.21.15.151:0/3630004042 shutdown_connections 2023-11-02T08:36:34.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.345+0000 7f85f2c1c640 1 -- 172.21.15.151:0/3630004042 wait complete. 2023-11-02T08:36:34.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.346+0000 7f85f2c1c640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.371 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.346+0000 7f85f2c1c640 10 monclient: monmap: 2023-11-02T08:36:34.371 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.371 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:34.372 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:34.372 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:34.372 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.372 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.372 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:34.373 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:34.373 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:34.373 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.373 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.346+0000 7f85f2c1c640 1 librados: starting msgr at 2023-11-02T08:36:34.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.346+0000 7f85f2c1c640 1 librados: starting objecter 2023-11-02T08:36:34.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.346+0000 7f85f2c1c640 1 Processor -- start 2023-11-02T08:36:34.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.346+0000 7f85f2c1c640 1 -- start start 2023-11-02T08:36:34.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.346+0000 7f85f2c1c640 1 librados: setting wanted keys 2023-11-02T08:36:34.374 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.346+0000 7f85f2c1c640 1 librados: calling monclient init 2023-11-02T08:36:34.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.346+0000 7f85f2c1c640 10 monclient: init 2023-11-02T08:36:34.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-02T08:36:34.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 0x7f85ec1d9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient(hunting): picked mon.noname-b con 0x7f85ec0b0ed0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.375 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f85ec0b1cb0 0x7f85ec0ac700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient(hunting): picked mon.noname-c con 0x7f85ec0b1cb0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85ec0b4570 0x7f85ec0acc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient(hunting): picked mon.noname-a con 0x7f85ec0b4570 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.376 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec1d9a30 con 0x7f85ec0b4570 2023-11-02T08:36:34.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec0ab070 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec0abb60 con 0x7f85ec0b1cb0 2023-11-02T08:36:34.377 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.347+0000 7f85f2c1c640 10 monclient(hunting): authenticate will time out at 1365.703464s 2023-11-02T08:36:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f09b8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 0x7f85ec1d9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f11b9640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85ec0b4570 0x7f85ec0acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f09b8640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 0x7f85ec1d9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36400/0 (socket says 172.21.15.151:36400) 2023-11-02T08:36:34.378 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f09b8640 1 -- 172.21.15.151:0/1120937566 learned_addr learned my addr 172.21.15.151:0/1120937566 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f09b8640 10 monclient(hunting): get_auth_request con 0x7f85ec0b0ed0 auth_method 0 2023-11-02T08:36:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f09b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f09b8640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f09b8640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.379 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f11b9640 10 monclient(hunting): get_auth_request con 0x7f85ec0b4570 auth_method 0 2023-11-02T08:36:34.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f11b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f11b9640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.348+0000 7f85f11b9640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.349+0000 7f85f09b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.380 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.349+0000 7f85f09b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.349+0000 7f85f09b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.349+0000 7f85f11b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.349+0000 7f85f11b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.349+0000 7f85f11b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.351+0000 7f85f09b8640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2023-11-02T08:36:34.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.351+0000 7f85f09b8640 1 -- 172.21.15.151:0/1120937566 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f85ec0b1cb0 msgr2=0x7f85ec0ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.351+0000 7f85f09b8640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f85ec0b1cb0 0x7f85ec0ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.351+0000 7f85f09b8640 1 -- 172.21.15.151:0/1120937566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85ec0b4570 msgr2=0x7f85ec0acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.351+0000 7f85f09b8640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85ec0b4570 0x7f85ec0acc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.382 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.351+0000 7f85f09b8640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.351+0000 7f85f09b8640 1 monclient: found mon.noname-b 2023-11-02T08:36:34.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.351+0000 7f85f09b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.352+0000 7f85f09b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.383 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.352+0000 7f85f09b8640 1 -- 172.21.15.151:0/1120937566 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80093a0 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.354+0000 7f85f09b8640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 0x7f85ec1d9e80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f85e000d5f0 tx=0x7f85e000dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.354+0000 7f85e97fa640 10 client.?.objecter ms_handle_connect 0x7f85ec0b0ed0 2023-11-02T08:36:34.387 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.354+0000 7f85e97fa640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:34.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 1 -- 172.21.15.151:0/1120937566 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e00125f0 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:34.388 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 10 monclient: dump: 2023-11-02T08:36:34.389 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.389 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.389 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.390 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.390 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.390 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.390 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.391 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.391 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.391 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.392 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 1 -- 172.21.15.151:0/1120937566 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e0012750 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85f2c1c640 5 monclient: authenticate success, global_id 4369 2023-11-02T08:36:34.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85f2c1c640 10 monclient: _renew_subs 2023-11-02T08:36:34.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85f2c1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec140f90 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.393 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 1 -- 172.21.15.151:0/1120937566 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e0012b00 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:34.394 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.357+0000 7f85e97fa640 10 monclient: dump: 2023-11-02T08:36:34.395 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.395 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.395 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.395 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.395 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.396 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.396 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.396 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.396 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.396 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.396 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85f2c1c640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:34.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85f2c1c640 10 monclient: _renew_subs 2023-11-02T08:36:34.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85f2c1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec1da5a0 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.397 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85f2c1c640 1 librados: init done 2023-11-02T08:36:34.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85eaffd640 4 set_mon_vals no callback set 2023-11-02T08:36:34.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85f2c1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.398 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85f2c1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85f2c1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.399 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.358+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec1d9800 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.409 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.405+0000 7fdeaee0e640 10 monclient: get_monmap_and_config 2023-11-02T08:36:34.409 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.405+0000 7fdeaee0e640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.405+0000 7fdeaee0e640 10 monclient: monmap: 2023-11-02T08:36:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.411 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:34.406681+0000 2023-11-02T08:36:34.411 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:34.406681+0000 2023-11-02T08:36:34.411 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.412 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.412 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:34.412 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:34.413 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:34.414 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.417 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 1 Processor -- start 2023-11-02T08:36:34.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 1 -- start start 2023-11-02T08:36:34.418 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 10 monclient: init 2023-11-02T08:36:34.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-11-02T08:36:34.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdea80b4570 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:34.419 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fdea80b0ed0 0x7fdea8144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 10 monclient(hunting): picked mon.noname-c con 0x7fdea80b0ed0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea813e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 10 monclient(hunting): picked mon.noname-b con 0x7fdea813e520 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea80a9780 con 0x7fdea80b4570 2023-11-02T08:36:34.421 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea80b4950 con 0x7fdea80b0ed0 2023-11-02T08:36:34.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.422 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.406+0000 7fdeaee0e640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea80aa1d0 con 0x7fdea813e520 2023-11-02T08:36:34.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.407+0000 7fdeaee0e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.407+0000 7fdeaee0e640 10 monclient(hunting): authenticate will time out at 1365.762877s 2023-11-02T08:36:34.423 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.407+0000 7fdead3ab640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea813e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.424 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.408+0000 7fdead3ab640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea813e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54932/0 (socket says 172.21.15.151:54932) 2023-11-02T08:36:34.424 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.408+0000 7fdead3ab640 1 -- 172.21.15.151:0/2909268066 learned_addr learned my addr 172.21.15.151:0/2909268066 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.424 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.408+0000 7fdead3ab640 10 monclient(hunting): get_auth_request con 0x7fdea813e520 auth_method 0 2023-11-02T08:36:34.425 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.408+0000 7fdead3ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.425 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.408+0000 7fdead3ab640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.425 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.408+0000 7fdead3ab640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.426 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.408+0000 7fdead3ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.426 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.408+0000 7fdead3ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.426 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.408+0000 7fdead3ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.427 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.410+0000 7fdead3ab640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2023-11-02T08:36:34.427 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.410+0000 7fdead3ab640 1 -- 172.21.15.151:0/2909268066 >> v2:172.21.15.178:3300/0 conn(0x7fdea80b0ed0 msgr2=0x7fdea8144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.427 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.410+0000 7fdead3ab640 1 --2- 172.21.15.151:0/2909268066 >> v2:172.21.15.178:3300/0 conn(0x7fdea80b0ed0 0x7fdea8144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.427 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.410+0000 7fdead3ab640 1 -- 172.21.15.151:0/2909268066 >> v1:172.21.15.151:6789/0 conn(0x7fdea80b4570 legacy=0x7fdea80a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.427 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.410+0000 7fdead3ab640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.427 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.410+0000 7fdead3ab640 1 monclient: found mon.noname-b 2023-11-02T08:36:34.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.410+0000 7fdead3ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.410+0000 7fdead3ab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.410+0000 7fdead3ab640 1 -- 172.21.15.151:0/2909268066 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea80aa4b0 con 0x7fdea813e520 2023-11-02T08:36:34.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.412+0000 7fdead3ab640 1 --2- 172.21.15.151:0/2909268066 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea813e9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009e80 tx=0x7fdea002bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc48d4e2b20080f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-02T08:36:34.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 1 -- 172.21.15.151:0/2909268066 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea002c930 con 0x7fdea813e520 2023-11-02T08:36:34.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:34.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 10 monclient: dump: 2023-11-02T08:36:34.434 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.434 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.434 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.434 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.435 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.435 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.435 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.435 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.435 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.436 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.436 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 1 -- 172.21.15.151:0/2909268066 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea002ca90 con 0x7fdea813e520 2023-11-02T08:36:34.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdeaee0e640 5 monclient: authenticate success, global_id 4286 2023-11-02T08:36:34.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdeaee0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:34.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdeaee0e640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:34.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdeaee0e640 4 set_mon_vals no callback set 2023-11-02T08:36:34.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 1 -- 172.21.15.151:0/2909268066 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea002ce20 con 0x7fdea813e520 2023-11-02T08:36:34.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:34.438 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.417+0000 7fdea77fe640 10 monclient: dump: 2023-11-02T08:36:34.438 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.439 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.439 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.439 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.439 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.439 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.440 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.440 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.440 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.440 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 10 monclient: shutdown 2023-11-02T08:36:34.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:34.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2909268066 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 msgr2=0x7fdea813e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 1 --2- 172.21.15.151:0/2909268066 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea813e9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009e80 tx=0x7fdea002bd90 comp rx=0 tx=0).stop 2023-11-02T08:36:34.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2909268066 shutdown_connections 2023-11-02T08:36:34.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 1 --2- 172.21.15.151:0/2909268066 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea813e9b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 1 --2- 172.21.15.151:0/2909268066 >> v2:172.21.15.178:3300/0 conn(0x7fdea80b0ed0 0x7fdea8144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2909268066 >> 172.21.15.151:0/2909268066 conn(0x7fdea8065d50 msgr2=0x7fdea8066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:34.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2909268066 shutdown_connections 2023-11-02T08:36:34.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.418+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2909268066 wait complete. 2023-11-02T08:36:34.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.419+0000 7fdeaee0e640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.419+0000 7fdeaee0e640 10 monclient: monmap: 2023-11-02T08:36:34.443 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.443 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:34.443 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:34.443 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:34.443 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.444 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.444 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:34.444 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:34.444 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:34.444 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.419+0000 7fdeaee0e640 1 librados: starting msgr at 2023-11-02T08:36:34.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.419+0000 7fdeaee0e640 1 librados: starting objecter 2023-11-02T08:36:34.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.419+0000 7fdeaee0e640 1 Processor -- start 2023-11-02T08:36:34.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.419+0000 7fdeaee0e640 1 -- start start 2023-11-02T08:36:34.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.419+0000 7fdeaee0e640 1 librados: setting wanted keys 2023-11-02T08:36:34.445 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.419+0000 7fdeaee0e640 1 librados: calling monclient init 2023-11-02T08:36:34.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.419+0000 7fdeaee0e640 10 monclient: init 2023-11-02T08:36:34.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:36:34.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdea80b0ed0 0x7fdea8141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient(hunting): picked mon.noname-a con 0x7fdea80b0ed0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.447 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdea80b4570 0x7fdea8141910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.447 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient(hunting): picked mon.noname-b con 0x7fdea80b4570 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.447 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea81d9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.447 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient(hunting): picked mon.noname-c con 0x7fdea813e520 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.447 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea81421c0 con 0x7fdea80b0ed0 2023-11-02T08:36:34.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea80b8100 con 0x7fdea80b4570 2023-11-02T08:36:34.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea8141ed0 con 0x7fdea813e520 2023-11-02T08:36:34.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdeaee0e640 10 monclient(hunting): authenticate will time out at 1365.776285s 2023-11-02T08:36:34.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.420+0000 7fdead3ab640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea81d9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.421+0000 7fdead3ab640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea81d9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54934/0 (socket says 172.21.15.151:54934) 2023-11-02T08:36:34.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.421+0000 7fdead3ab640 1 -- 172.21.15.151:0/2086077792 learned_addr learned my addr 172.21.15.151:0/2086077792 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.421+0000 7fdead3ab640 10 monclient(hunting): get_auth_request con 0x7fdea813e520 auth_method 0 2023-11-02T08:36:34.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.421+0000 7fdead3ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.421+0000 7fdead3ab640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.421+0000 7fdead3ab640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.421+0000 7fdead3ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.421+0000 7fdead3ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.421+0000 7fdead3ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2023-11-02T08:36:34.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 1 -- 172.21.15.151:0/2086077792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdea80b4570 msgr2=0x7fdea8141910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdea80b4570 0x7fdea8141910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 1 -- 172.21.15.151:0/2086077792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdea80b0ed0 msgr2=0x7fdea8141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdea80b0ed0 0x7fdea8141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 1 monclient: found mon.noname-c 2023-11-02T08:36:34.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.424+0000 7fdead3ab640 1 -- 172.21.15.151:0/2086077792 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea0009b30 con 0x7fdea813e520 2023-11-02T08:36:34.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.427+0000 7fdead3ab640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea81d9bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009fa0 tx=0x7fdea0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.427+0000 7fdea57fa640 10 client.?.objecter ms_handle_connect 0x7fdea813e520 2023-11-02T08:36:34.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.427+0000 7fdea57fa640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:34.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 1 -- 172.21.15.151:0/2086077792 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0009510 con 0x7fdea813e520 2023-11-02T08:36:34.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:34.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 10 monclient: dump: 2023-11-02T08:36:34.456 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.456 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.456 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.456 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.457 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.457 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.457 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.457 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.457 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.458 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 1 -- 172.21.15.151:0/2086077792 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea003e450 con 0x7fdea813e520 2023-11-02T08:36:34.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdeaee0e640 5 monclient: authenticate success, global_id 4289 2023-11-02T08:36:34.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdeaee0e640 10 monclient: _renew_subs 2023-11-02T08:36:34.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdeaee0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea81426e0 con 0x7fdea813e520 2023-11-02T08:36:34.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 1 -- 172.21.15.151:0/2086077792 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea003e7c0 con 0x7fdea813e520 2023-11-02T08:36:34.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:34.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea57fa640 10 monclient: dump: 2023-11-02T08:36:34.460 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.460 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.461 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.461 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.461 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.461 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.461 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.462 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.462 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.462 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdea6ffd640 4 set_mon_vals no callback set 2023-11-02T08:36:34.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdeaee0e640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:34.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdeaee0e640 10 monclient: _renew_subs 2023-11-02T08:36:34.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdeaee0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea81da130 con 0x7fdea813e520 2023-11-02T08:36:34.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.432+0000 7fdeaee0e640 1 librados: init done 2023-11-02T08:36:34.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.433+0000 7fdeaee0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.433+0000 7fdeaee0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.433+0000 7fdeaee0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.433+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea8067920 con 0x7fdea813e520 2023-11-02T08:36:34.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.453+0000 7faf476ca640 10 monclient: get_monmap_and_config 2023-11-02T08:36:34.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.453+0000 7faf476ca640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.453+0000 7faf476ca640 10 monclient: monmap: 2023-11-02T08:36:34.467 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.467 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.467 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:34.455012+0000 2023-11-02T08:36:34.467 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:34.455012+0000 2023-11-02T08:36:34.468 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.468 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.468 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:34.468 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:34.468 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:34.469 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:34.469 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:34.469 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.454+0000 7faf476ca640 1 Processor -- start 2023-11-02T08:36:34.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.454+0000 7faf476ca640 1 -- start start 2023-11-02T08:36:34.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.454+0000 7faf476ca640 10 monclient: init 2023-11-02T08:36:34.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-11-02T08:36:34.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7faf40069ee0 0x7faf400af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 10 monclient(hunting): picked mon.noname-a con 0x7faf40069ee0 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:34.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faf4013e7f0 0x7faf4013ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 10 monclient(hunting): picked mon.noname-b con 0x7faf4013e7f0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf401419d0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:34.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf400ae3a0 con 0x7faf401419d0 2023-11-02T08:36:34.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf4006a2c0 con 0x7faf40069ee0 2023-11-02T08:36:34.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf40142570 con 0x7faf4013e7f0 2023-11-02T08:36:34.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.455+0000 7faf476ca640 10 monclient(hunting): authenticate will time out at 1365.811561s 2023-11-02T08:36:34.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.456+0000 7faf45466640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7faf40069ee0 0x7faf400af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.456+0000 7faf45466640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7faf40069ee0 0x7faf400af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38728/0 (socket says 172.21.15.151:38728) 2023-11-02T08:36:34.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.456+0000 7faf45466640 1 -- 172.21.15.151:0/3563738358 learned_addr learned my addr 172.21.15.151:0/3563738358 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.456+0000 7faf45466640 10 monclient(hunting): get_auth_request con 0x7faf40069ee0 auth_method 0 2023-11-02T08:36:34.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.456+0000 7faf45466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.456+0000 7faf45466640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.456+0000 7faf45466640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.457+0000 7faf45466640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.457+0000 7faf45466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.457+0000 7faf45466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.459+0000 7faf45466640 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2023-11-02T08:36:34.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.459+0000 7faf45466640 1 -- 172.21.15.151:0/3563738358 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faf4013e7f0 msgr2=0x7faf4013ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.459+0000 7faf45466640 1 --2- 172.21.15.151:0/3563738358 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faf4013e7f0 0x7faf4013ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.459+0000 7faf45466640 1 -- 172.21.15.151:0/3563738358 >> v1:172.21.15.178:6789/0 conn(0x7faf401419d0 legacy=0x7faf40141e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.459+0000 7faf45466640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.459+0000 7faf45466640 1 monclient: found mon.noname-a 2023-11-02T08:36:34.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.459+0000 7faf45466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.459+0000 7faf45466640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.459+0000 7faf45466640 1 -- 172.21.15.151:0/3563738358 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf400af810 con 0x7faf40069ee0 2023-11-02T08:36:34.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.461+0000 7faf45466640 1 --2- 172.21.15.151:0/3563738358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 0x7faf400af1c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009870 tx=0x7faf3c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fd900c3c037854 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.480 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 1 -- 172.21.15.151:0/3563738358 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c031740 con 0x7faf40069ee0 2023-11-02T08:36:34.480 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:34.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 10 monclient: dump: 2023-11-02T08:36:34.481 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.481 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.481 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.482 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.482 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.482 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.482 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.482 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.483 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.483 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 1 -- 172.21.15.151:0/3563738358 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3c0318a0 con 0x7faf40069ee0 2023-11-02T08:36:34.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 1 -- 172.21.15.151:0/3563738358 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c031c30 con 0x7faf40069ee0 2023-11-02T08:36:34.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:34.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf37fff640 10 monclient: dump: 2023-11-02T08:36:34.485 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.485 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.485 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.485 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.485 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.486 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.486 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.486 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.486 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.486 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf476ca640 5 monclient: authenticate success, global_id 4320 2023-11-02T08:36:34.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf476ca640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:34.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.465+0000 7faf476ca640 4 set_mon_vals no callback set 2023-11-02T08:36:34.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 10 monclient: shutdown 2023-11-02T08:36:34.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:34.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 1 -- 172.21.15.151:0/3563738358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 msgr2=0x7faf400af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 1 --2- 172.21.15.151:0/3563738358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 0x7faf400af1c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009870 tx=0x7faf3c02b7e0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 1 -- 172.21.15.151:0/3563738358 shutdown_connections 2023-11-02T08:36:34.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 1 --2- 172.21.15.151:0/3563738358 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faf4013e7f0 0x7faf4013ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 1 --2- 172.21.15.151:0/3563738358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 0x7faf400af1c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 1 -- 172.21.15.151:0/3563738358 >> 172.21.15.151:0/3563738358 conn(0x7faf400b3fa0 msgr2=0x7faf400b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:34.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 1 -- 172.21.15.151:0/3563738358 shutdown_connections 2023-11-02T08:36:34.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.466+0000 7faf476ca640 1 -- 172.21.15.151:0/3563738358 wait complete. 2023-11-02T08:36:34.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.467+0000 7faf476ca640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.467+0000 7faf476ca640 10 monclient: monmap: 2023-11-02T08:36:34.489 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.489 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:34.490 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:34.490 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:34.490 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.490 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.490 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:34.491 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:34.491 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:34.491 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.468+0000 7faf476ca640 1 librados: starting msgr at 2023-11-02T08:36:34.491 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.468+0000 7faf476ca640 1 librados: starting objecter 2023-11-02T08:36:34.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.468+0000 7faf476ca640 1 Processor -- start 2023-11-02T08:36:34.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.468+0000 7faf476ca640 1 -- start start 2023-11-02T08:36:34.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.468+0000 7faf476ca640 1 librados: setting wanted keys 2023-11-02T08:36:34.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.468+0000 7faf476ca640 1 librados: calling monclient init 2023-11-02T08:36:34.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.468+0000 7faf476ca640 10 monclient: init 2023-11-02T08:36:34.492 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:36:34.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 0x7faf401ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient(hunting): picked mon.noname-a con 0x7faf40069ee0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4013e7f0 0x7faf4013ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient(hunting): picked mon.noname-b con 0x7faf4013e7f0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faf401419d0 0x7faf4013c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient(hunting): picked mon.noname-c con 0x7faf401419d0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4013e0e0 con 0x7faf40069ee0 2023-11-02T08:36:34.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4013faa0 con 0x7faf4013e7f0 2023-11-02T08:36:34.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4006e790 con 0x7faf401419d0 2023-11-02T08:36:34.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf476ca640 10 monclient(hunting): authenticate will time out at 1365.825110s 2023-11-02T08:36:34.495 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.469+0000 7faf45466640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 0x7faf401ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.470+0000 7faf45466640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 0x7faf401ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38736/0 (socket says 172.21.15.151:38736) 2023-11-02T08:36:34.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.470+0000 7faf45466640 1 -- 172.21.15.151:0/3515246389 learned_addr learned my addr 172.21.15.151:0/3515246389 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.470+0000 7faf45466640 10 monclient(hunting): get_auth_request con 0x7faf40069ee0 auth_method 0 2023-11-02T08:36:34.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.470+0000 7faf45466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.496 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.470+0000 7faf45466640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.470+0000 7faf45466640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.470+0000 7faf45466640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.470+0000 7faf45466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.470+0000 7faf45466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.473+0000 7faf45466640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2023-11-02T08:36:34.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.473+0000 7faf45466640 1 -- 172.21.15.151:0/3515246389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faf401419d0 msgr2=0x7faf4013c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.473+0000 7faf45466640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faf401419d0 0x7faf4013c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.473+0000 7faf45466640 1 -- 172.21.15.151:0/3515246389 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4013e7f0 msgr2=0x7faf4013ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.473+0000 7faf45466640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4013e7f0 0x7faf4013ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.473+0000 7faf45466640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.473+0000 7faf45466640 1 monclient: found mon.noname-a 2023-11-02T08:36:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.473+0000 7faf45466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.474+0000 7faf45466640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.474+0000 7faf45466640 1 -- 172.21.15.151:0/3515246389 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c0094d0 con 0x7faf40069ee0 2023-11-02T08:36:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.476+0000 7faf45466640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 0x7faf401ca140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf3c000c00 tx=0x7faf3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.476+0000 7faf35ffb640 10 client.?.objecter ms_handle_connect 0x7faf40069ee0 2023-11-02T08:36:34.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.476+0000 7faf35ffb640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:34.501 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.479+0000 7faf35ffb640 1 -- 172.21.15.151:0/3515246389 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c033c70 con 0x7faf40069ee0 2023-11-02T08:36:34.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:34.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 10 monclient: dump: 2023-11-02T08:36:34.502 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.502 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.503 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.503 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.503 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.503 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.503 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.504 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.504 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.504 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 1 -- 172.21.15.151:0/3515246389 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3c033df0 con 0x7faf40069ee0 2023-11-02T08:36:34.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 1 -- 172.21.15.151:0/3515246389 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c03d6b0 con 0x7faf40069ee0 2023-11-02T08:36:34.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:34.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf35ffb640 10 monclient: dump: 2023-11-02T08:36:34.506 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.506 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.506 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.506 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.507 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.507 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.507 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.507 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.507 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.508 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf476ca640 5 monclient: authenticate success, global_id 4323 2023-11-02T08:36:34.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf377fe640 4 set_mon_vals no callback set 2023-11-02T08:36:34.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf476ca640 10 monclient: _renew_subs 2023-11-02T08:36:34.508 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf476ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4013c7c0 con 0x7faf40069ee0 2023-11-02T08:36:34.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf476ca640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:34.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf476ca640 10 monclient: _renew_subs 2023-11-02T08:36:34.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf476ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf401414f0 con 0x7faf40069ee0 2023-11-02T08:36:34.509 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.480+0000 7faf476ca640 1 librados: init done 2023-11-02T08:36:34.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.481+0000 7faf476ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.481+0000 7faf476ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.481+0000 7faf476ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.510 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.481+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf40067920 con 0x7faf40069ee0 2023-11-02T08:36:34.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.500+0000 7faf35ffb640 1 -- 172.21.15.151:0/3515246389 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7faf3c03a040 con 0x7faf40069ee0 2023-11-02T08:36:34.511 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.501+0000 7faf35ffb640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7faf2406bdc0 0x7faf2406e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.501+0000 7faf44c65640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7faf2406bdc0 0x7faf2406e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.513 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.501+0000 7faf44c65640 10 monclient: get_auth_request con 0x7faf2406bdc0 auth_method 0 2023-11-02T08:36:34.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.502+0000 7faf35ffb640 10 client.4323.objecter ms_handle_connect 0x7faf2406bdc0 2023-11-02T08:36:34.514 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.502+0000 7faf44c65640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7faf2406bdc0 0x7faf2406e280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faf4013e3e0 tx=0x7faf30006c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-02T08:36:34.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.537+0000 7f9714bab640 10 monclient: get_monmap_and_config 2023-11-02T08:36:34.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.537+0000 7f9714bab640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.541 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.537+0000 7f9714bab640 10 monclient: monmap: 2023-11-02T08:36:34.541 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.542 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.542 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:34.538393+0000 2023-11-02T08:36:34.542 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:34.538393+0000 2023-11-02T08:36:34.542 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.542 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.542 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:34.543 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:34.543 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:34.543 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:34.543 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:34.543 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.537+0000 7f9714bab640 1 Processor -- start 2023-11-02T08:36:34.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.537+0000 7f9714bab640 1 -- start start 2023-11-02T08:36:34.544 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.537+0000 7f9714bab640 10 monclient: init 2023-11-02T08:36:34.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-11-02T08:36:34.545 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f97100b4570 0x7f9710144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient(hunting): picked mon.noname-c con 0x7f97100b4570 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:34.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f97100b0ed0 0x7f97100a9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.546 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient(hunting): picked mon.noname-a con 0x7f97100b0ed0 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:34.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97100b1cb0 0x7f97100b2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient(hunting): picked mon.noname-b con 0x7f97100b1cb0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97100b4950 con 0x7f97100b0ed0 2023-11-02T08:36:34.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97100aa1d0 con 0x7f97100b4570 2023-11-02T08:36:34.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97100aa4b0 con 0x7f97100b1cb0 2023-11-02T08:36:34.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.538+0000 7f9714bab640 10 monclient(hunting): authenticate will time out at 1365.894687s 2023-11-02T08:36:34.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-02T08:36:34.606 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.602+0000 7f85e97fa640 1 -- 172.21.15.151:0/1120937566 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f85e0012c60 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.603+0000 7f85e97fa640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f85d006be10 0x7f85d006e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.603+0000 7f85ebfff640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f85d006be10 0x7f85d006e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.607 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.603+0000 7f85ebfff640 10 monclient: get_auth_request con 0x7f85d006be10 auth_method 0 2023-11-02T08:36:34.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.604+0000 7f85e97fa640 10 client.4369.objecter ms_handle_connect 0x7f85d006be10 2023-11-02T08:36:34.608 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.604+0000 7f85ebfff640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f85d006be10 0x7f85d006e2d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f85d8036000 tx=0x7f85d8030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.606+0000 7f85e97fa640 1 -- 172.21.15.151:0/1120937566 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f85e0059040 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.606+0000 7f85e97fa640 10 client.4369.objecter ms_dispatch 0x7f85ec000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:34.609 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.606+0000 7f85e97fa640 3 client.4369.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:34.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.606+0000 7f85e97fa640 3 client.4369.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:34.610 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.606+0000 7f85e97fa640 20 client.4369.objecter dump_active .. 0 homeless 2023-11-02T08:36:34.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-02T08:36:34.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.616+0000 7f85e97fa640 1 -- 172.21.15.151:0/1120937566 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f85e0054c80 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.619 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.616+0000 7f85e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.620 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.616+0000 7f85e97fa640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:34.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.640+0000 7fc602c74640 10 monclient: get_monmap_and_config 2023-11-02T08:36:34.643 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.640+0000 7fc602c74640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.644 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.640+0000 7fc602c74640 10 monclient: monmap: 2023-11-02T08:36:34.644 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.644 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.644 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:34.641234+0000 2023-11-02T08:36:34.644 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:34.641234+0000 2023-11-02T08:36:34.645 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.645 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.645 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:34.645 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:34.645 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:34.646 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:34.646 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:34.646 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-02T08:36:34.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.640+0000 7fc602c74640 1 Processor -- start 2023-11-02T08:36:34.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.640+0000 7fc602c74640 1 -- start start 2023-11-02T08:36:34.648 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.640+0000 7fc602c74640 10 monclient: init 2023-11-02T08:36:34.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-11-02T08:36:34.649 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5fc0b4570 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:34.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5fc0b0ed0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:34.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc5fc144d40 0x7fc5fc13ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.650 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 10 monclient(hunting): picked mon.noname-a con 0x7fc5fc144d40 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:34.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5fc0a9780 con 0x7fc5fc0b4570 2023-11-02T08:36:34.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5fc1451d0 con 0x7fc5fc0b0ed0 2023-11-02T08:36:34.651 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5fc0aa1d0 con 0x7fc5fc144d40 2023-11-02T08:36:34.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.652 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.641+0000 7fc602c74640 10 monclient(hunting): authenticate will time out at 1365.997674s 2023-11-02T08:36:34.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.657+0000 7faf35ffb640 1 -- 172.21.15.151:0/3515246389 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faf3c07a690 con 0x7faf40069ee0 2023-11-02T08:36:34.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.657+0000 7faf35ffb640 10 client.4323.objecter ms_dispatch 0x7faf40000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:34.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.657+0000 7faf35ffb640 3 client.4323.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:34.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.657+0000 7faf35ffb640 3 client.4323.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:34.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.657+0000 7faf35ffb640 20 client.4323.objecter dump_active .. 0 homeless 2023-11-02T08:36:34.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.669+0000 7fc5fbfff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fc5fc0b0ed0 0x7fc5fc13e520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:42916/0 (socket says 172.21.15.151:42916) 2023-11-02T08:36:34.672 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.669+0000 7fc5fbfff640 1 -- 172.21.15.151:0/1641311213 learned_addr learned my addr 172.21.15.151:0/1641311213 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:34.675 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.671+0000 7f970e59c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f97100b4570 0x7f9710144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260149708 0 0) 0x7fc5fc1451d0 con 0x7fc5fc0b0ed0 2023-11-02T08:36:34.677 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7fc5fb7fe640 10 monclient(hunting): _negotiate 2023-11-02T08:36:34.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7fc5fb7fe640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7fc5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7fc5fb7fe640 10 monclient(hunting): my global_id is 4372 2023-11-02T08:36:34.678 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ec003520 con 0x7fc5fc0b0ed0 2023-11-02T08:36:34.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7f970e59c640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f97100b4570 0x7f9710144d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36446/0 (socket says 172.21.15.151:36446) 2023-11-02T08:36:34.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7f970e59c640 1 -- 172.21.15.151:0/2888594357 learned_addr learned my addr 172.21.15.151:0/2888594357 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7f970e59c640 10 monclient(hunting): get_auth_request con 0x7f97100b4570 auth_method 0 2023-11-02T08:36:34.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7f970e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.679 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7f970e59c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.674+0000 7f970e59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.675+0000 7f970e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.675+0000 7f970e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.680 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.675+0000 7f970e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.681 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2023-11-02T08:36:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 1 -- 172.21.15.151:0/2888594357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97100b1cb0 msgr2=0x7f97100b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 1 --2- 172.21.15.151:0/2888594357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97100b1cb0 0x7f97100b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 1 -- 172.21.15.151:0/2888594357 >> v2:172.21.15.151:3300/0 conn(0x7f97100b0ed0 msgr2=0x7f97100a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.682 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 1 --2- 172.21.15.151:0/2888594357 >> v2:172.21.15.151:3300/0 conn(0x7f97100b0ed0 0x7f97100a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 1 monclient: found mon.noname-c 2023-11-02T08:36:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.683 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.677+0000 7f970e59c640 1 -- 172.21.15.151:0/2888594357 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97100a8b50 con 0x7f97100b4570 2023-11-02T08:36:34.685 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1019331121 0 0) 0x7fc5ec003520 con 0x7fc5fc0b0ed0 2023-11-02T08:36:34.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 >> v2:172.21.15.151:3300/0 conn(0x7fc5fc144d40 msgr2=0x7fc5fc13ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 --2- 172.21.15.151:0/1641311213 >> v2:172.21.15.151:3300/0 conn(0x7fc5fc144d40 0x7fc5fc13ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 >> v1:172.21.15.151:6789/0 conn(0x7fc5fc0b4570 legacy=0x7fc5fc0a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.686 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 monclient: found mon.noname-c-legacy 2023-11-02T08:36:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.178:6789/0 2023-11-02T08:36:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc0aa4b0 con 0x7fc5fc0b0ed0 2023-11-02T08:36:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.687 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 650167734 0 0) 0x7fc5e80017f0 con 0x7fc5fc0b0ed0 2023-11-02T08:36:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-11-02T08:36:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient: dump: 2023-11-02T08:36:34.689 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.689 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.689 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.689 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.689 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.690 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.690 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.690 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.690 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.690 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.690 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] but i'm connected to v1:172.21.15.178:6789/0, reconnecting 2023-11-02T08:36:34.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 >> v1:172.21.15.178:6789/0 conn(0x7fc5fc0b0ed0 legacy=0x7fc5fc13e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-02T08:36:34.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 --2- 172.21.15.151:0/1641311213 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec003970 0x7fc5ec003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.691 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient(hunting): picked mon.c con 0x7fc5ec003970 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 --2- 172.21.15.151:0/1641311213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec0047c0 0x7fc5ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient(hunting): picked mon.a con 0x7fc5ec0047c0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 --2- 172.21.15.151:0/1641311213 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5ec005380 0x7fc5ec00b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient(hunting): picked mon.b con 0x7fc5ec005380 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.692 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc0aa1d0 con 0x7fc5ec0047c0 2023-11-02T08:36:34.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec00c2c0 con 0x7fc5ec005380 2023-11-02T08:36:34.693 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec00c480 con 0x7fc5ec003970 2023-11-02T08:36:34.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.694 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.682+0000 7fc5fb7fe640 10 monclient(hunting): _finish_auth 0 2023-11-02T08:36:34.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.683+0000 7f970e59c640 1 --2- 172.21.15.151:0/2888594357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b4570 0x7f9710144d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9700009870 tx=0x7f970002b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8af4996ad5387402 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 1 -- 172.21.15.151:0/2888594357 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700031740 con 0x7f97100b4570 2023-11-02T08:36:34.695 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 10 monclient: dump: 2023-11-02T08:36:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.696 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.697 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.697 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.697 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.697 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.697 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.697 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.698 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.698 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f9714bab640 5 monclient: authenticate success, global_id 4375 2023-11-02T08:36:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f9714bab640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 1 -- 172.21.15.151:0/2888594357 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97000318a0 con 0x7f97100b4570 2023-11-02T08:36:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f9714bab640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:34.699 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f9714bab640 4 set_mon_vals no callback set 2023-11-02T08:36:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 1 -- 172.21.15.151:0/2888594357 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700031740 con 0x7f97100b4570 2023-11-02T08:36:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.690+0000 7f970d59a640 10 monclient: dump: 2023-11-02T08:36:34.700 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.702 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.702 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.702 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.702 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.686+0000 7fc601211640 1 --2- 172.21.15.151:0/1641311213 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5ec005380 0x7fc5ec00b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.686+0000 7fc601211640 10 monclient(hunting): get_auth_request con 0x7fc5ec005380 auth_method 0 2023-11-02T08:36:34.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.686+0000 7fc601211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.686+0000 7fc601211640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.686+0000 7fc601211640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:36:34.703 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.689+0000 7fc601211640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.689+0000 7fc601211640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.704 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.689+0000 7fc601211640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-02T08:36:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 10 monclient: shutdown 2023-11-02T08:36:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 1 -- 172.21.15.151:0/2888594357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b4570 msgr2=0x7f9710144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 1 --2- 172.21.15.151:0/2888594357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b4570 0x7f9710144d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9700009870 tx=0x7f970002b7e0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.705 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 1 -- 172.21.15.151:0/2888594357 shutdown_connections 2023-11-02T08:36:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 1 --2- 172.21.15.151:0/2888594357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97100b1cb0 0x7f97100b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 1 --2- 172.21.15.151:0/2888594357 >> v2:172.21.15.151:3300/0 conn(0x7f97100b0ed0 0x7f97100a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 1 --2- 172.21.15.151:0/2888594357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b4570 0x7f9710144d40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.706 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 1 -- 172.21.15.151:0/2888594357 >> 172.21.15.151:0/2888594357 conn(0x7f9710065d50 msgr2=0x7f9710066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:34.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 1 -- 172.21.15.151:0/2888594357 shutdown_connections 2023-11-02T08:36:34.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.691+0000 7f9714bab640 1 -- 172.21.15.151:0/2888594357 wait complete. 2023-11-02T08:36:34.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 10 monclient(hunting): handle_auth_done global_id 4372 payload 293 2023-11-02T08:36:34.707 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 1 -- 172.21.15.151:0/1641311213 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec003970 msgr2=0x7fc5ec003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 1 --2- 172.21.15.151:0/1641311213 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5ec003970 0x7fc5ec003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 1 -- 172.21.15.151:0/1641311213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec0047c0 msgr2=0x7fc5ec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 1 --2- 172.21.15.151:0/1641311213 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec0047c0 0x7fc5ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.708 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 1 monclient: found mon.b 2023-11-02T08:36:34.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 1 -- 172.21.15.151:0/1641311213 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc0aa4b0 con 0x7fc5ec005380 2023-11-02T08:36:34.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.709 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.692+0000 7fc601211640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.693+0000 7f9714bab640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.710 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.693+0000 7f9714bab640 10 monclient: monmap: 2023-11-02T08:36:34.711 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.711 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:34.711 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:34.711 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:34.711 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.711 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.712 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:34.712 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:34.712 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:34.712 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.712 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.694+0000 7f9714bab640 1 librados: starting msgr at 2023-11-02T08:36:34.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.693+0000 7fc601211640 1 -- 172.21.15.151:0/1641311213 reap_dead start 2023-11-02T08:36:34.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.693+0000 7fc602c74640 5 monclient: authenticate success, global_id 4372 2023-11-02T08:36:34.713 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.693+0000 7fc602c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:34.720 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.694+0000 7f9714bab640 1 librados: starting objecter 2023-11-02T08:36:34.721 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.694+0000 7f9714bab640 1 Processor -- start 2023-11-02T08:36:34.721 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.694+0000 7f9714bab640 1 -- start start 2023-11-02T08:36:34.721 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.694+0000 7f9714bab640 1 librados: setting wanted keys 2023-11-02T08:36:34.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.694+0000 7f9714bab640 1 librados: calling monclient init 2023-11-02T08:36:34.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.694+0000 7f9714bab640 10 monclient: init 2023-11-02T08:36:34.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.695+0000 7f9714bab640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.722 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.695+0000 7f9714bab640 10 monclient: _add_conns ranks=[1,0,2] 2023-11-02T08:36:34.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.695+0000 7f9714bab640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b0ed0 0x7f97100ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.695+0000 7f9714bab640 10 monclient(hunting): picked mon.noname-b con 0x7f97100b0ed0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.696+0000 7f9714bab640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97100b1cb0 0x7f97100abf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.696+0000 7f9714bab640 10 monclient(hunting): picked mon.noname-a con 0x7f97100b1cb0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.696+0000 7f9714bab640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97100b4570 0x7f9710138650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.696+0000 7f970e59c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b0ed0 0x7f97100ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.696+0000 7f9714bab640 10 monclient(hunting): picked mon.noname-c con 0x7f97100b4570 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.696+0000 7f970e59c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b0ed0 0x7f97100ab4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36474/0 (socket says 172.21.15.151:36474) 2023-11-02T08:36:34.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.696+0000 7f970e59c640 1 -- 172.21.15.151:0/4262689732 learned_addr learned my addr 172.21.15.151:0/4262689732 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.696+0000 7f9714bab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97100ac670 con 0x7f97100b1cb0 2023-11-02T08:36:34.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f9714bab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.725 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97101d4270 con 0x7f97100b0ed0 2023-11-02T08:36:34.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f9714bab640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9710142330 con 0x7f97100b4570 2023-11-02T08:36:34.726 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f9714bab640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f9714bab640 10 monclient(hunting): authenticate will time out at 1366.053289s 2023-11-02T08:36:34.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f970e59c640 10 monclient(hunting): get_auth_request con 0x7f97100b0ed0 auth_method 0 2023-11-02T08:36:34.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f970e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f970e59c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.727 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.697+0000 7f970e59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.698+0000 7f970e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.698+0000 7f970e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.698+0000 7f970e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2023-11-02T08:36:34.728 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 1 -- 172.21.15.151:0/4262689732 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97100b4570 msgr2=0x7f9710138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97100b4570 0x7f9710138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 1 -- 172.21.15.151:0/4262689732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97100b1cb0 msgr2=0x7f97100abf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97100b1cb0 0x7f97100abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 1 monclient: found mon.noname-b 2023-11-02T08:36:34.729 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7f970e59c640 1 -- 172.21.15.151:0/4262689732 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97000094d0 con 0x7f97100b0ed0 2023-11-02T08:36:34.730 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.702+0000 7fc601211640 1 --2- 172.21.15.151:0/1641311213 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5ec005380 0x7fc5ec00b9e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009e40 tx=0x7fc5ec0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a7b967125b65099 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.704+0000 7f970e59c640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b0ed0 0x7f97100ab4c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f970002b820 tx=0x7f97000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.704+0000 7f96feffd640 10 client.?.objecter ms_handle_connect 0x7f97100b0ed0 2023-11-02T08:36:34.731 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.704+0000 7f96feffd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:34.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.705+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f402cd90 con 0x7fc5ec005380 2023-11-02T08:36:34.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.705+0000 7fc5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.734 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.705+0000 7fc5fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:34.735 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.705+0000 7fc5fb7fe640 10 monclient: dump: 2023-11-02T08:36:34.735 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.735 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.735 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.735 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.736 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.736 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.736 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.736 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.736 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.737 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.705+0000 7fc602c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:34.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.708+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f4003990 con 0x7fc5ec005380 2023-11-02T08:36:34.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.708+0000 7fc5fb7fe640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.737 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.708+0000 7fc5fb7fe640 1 -- 172.21.15.151:0/1641311213 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4003d40 con 0x7fc5ec005380 2023-11-02T08:36:34.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.708+0000 7fc5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.708+0000 7fc5fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:34.738 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.708+0000 7fc5fb7fe640 10 monclient: dump: 2023-11-02T08:36:34.738 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.738 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.739 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.739 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.739 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.739 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.739 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.740 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.740 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.740 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.708+0000 7fc602c74640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:34.740 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.708+0000 7fc602c74640 4 set_mon_vals no callback set 2023-11-02T08:36:34.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.709+0000 7fc602c74640 10 monclient: shutdown 2023-11-02T08:36:34.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.709+0000 7fc602c74640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:34.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.709+0000 7fc602c74640 1 -- 172.21.15.151:0/1641311213 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5ec005380 msgr2=0x7fc5ec00b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.709+0000 7fc602c74640 1 --2- 172.21.15.151:0/1641311213 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5ec005380 0x7fc5ec00b9e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009e40 tx=0x7fc5ec0057f0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.709+0000 7fc602c74640 1 -- 172.21.15.151:0/1641311213 shutdown_connections 2023-11-02T08:36:34.741 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.709+0000 7fc602c74640 1 --2- 172.21.15.151:0/1641311213 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5ec005380 0x7fc5ec00b9e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.709+0000 7fc602c74640 1 -- 172.21.15.151:0/1641311213 >> 172.21.15.151:0/1641311213 conn(0x7fc5fc065d50 msgr2=0x7fc5fc066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:34.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.709+0000 7fc602c74640 1 -- 172.21.15.151:0/1641311213 shutdown_connections 2023-11-02T08:36:34.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.709+0000 7fc602c74640 1 -- 172.21.15.151:0/1641311213 wait complete. 2023-11-02T08:36:34.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.710+0000 7fc602c74640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.742 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.710+0000 7fc602c74640 10 monclient: monmap: 2023-11-02T08:36:34.743 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.743 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:34.743 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:34.743 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:34.743 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.744 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.744 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:34.744 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:34.744 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:34.744 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.711+0000 7fc602c74640 1 librados: starting msgr at 2023-11-02T08:36:34.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.711+0000 7fc602c74640 1 librados: starting objecter 2023-11-02T08:36:34.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.711+0000 7fc602c74640 1 Processor -- start 2023-11-02T08:36:34.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.711+0000 7fc602c74640 1 -- start start 2023-11-02T08:36:34.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.711+0000 7fc602c74640 1 librados: setting wanted keys 2023-11-02T08:36:34.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.711+0000 7fc602c74640 1 librados: calling monclient init 2023-11-02T08:36:34.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.711+0000 7fc602c74640 10 monclient: init 2023-11-02T08:36:34.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.712+0000 7fc602c74640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.712+0000 7fc602c74640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:36:34.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.712+0000 7fc602c74640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec005380 0x7fc5fc0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.746 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ec005380 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5fc0ab560 0x7fc5fc0ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 1 -- 172.21.15.151:0/4262689732 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700031740 con 0x7f97100b0ed0 2023-11-02T08:36:34.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.747 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:34.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 10 monclient: dump: 2023-11-02T08:36:34.748 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.748 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.748 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.748 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.749 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.749 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.749 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.749 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.749 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.750 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.750 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 1 -- 172.21.15.151:0/4262689732 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9700031ef0 con 0x7f97100b0ed0 2023-11-02T08:36:34.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 1 -- 172.21.15.151:0/4262689732 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970003d470 con 0x7f97100b0ed0 2023-11-02T08:36:34.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:34.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f96feffd640 10 monclient: dump: 2023-11-02T08:36:34.751 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.752 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.752 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.752 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.752 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.752 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.753 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.753 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.753 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.753 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 10 monclient(hunting): picked mon.noname-c con 0x7fc5fc0ab560 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5fc1d6260 0x7fc5fc1d66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 10 monclient(hunting): picked mon.noname-b con 0x7fc5fc1d6260 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc1d6c20 con 0x7fc5ec005380 2023-11-02T08:36:34.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc1424e0 con 0x7fc5fc1d6260 2023-11-02T08:36:34.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc1427c0 con 0x7fc5fc0ab560 2023-11-02T08:36:34.756 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc602c74640 10 monclient(hunting): authenticate will time out at 1366.069695s 2023-11-02T08:36:34.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7fc601211640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5fc1d6260 0x7fc5fc1d66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.713+0000 7f970cd99640 4 set_mon_vals no callback set 2023-11-02T08:36:34.757 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7f9714bab640 5 monclient: authenticate success, global_id 4378 2023-11-02T08:36:34.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7f9714bab640 10 monclient: _renew_subs 2023-11-02T08:36:34.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7f9714bab640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9710138b90 con 0x7f97100b0ed0 2023-11-02T08:36:34.758 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7f9714bab640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:34.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7f9714bab640 10 monclient: _renew_subs 2023-11-02T08:36:34.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7f9714bab640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9710139050 con 0x7f97100b0ed0 2023-11-02T08:36:34.759 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7f9714bab640 1 librados: init done 2023-11-02T08:36:34.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7fc601211640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5fc1d6260 0x7fc5fc1d66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36484/0 (socket says 172.21.15.151:36484) 2023-11-02T08:36:34.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7fc601211640 1 -- 172.21.15.151:0/3474826060 learned_addr learned my addr 172.21.15.151:0/3474826060 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.760 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7fc601211640 10 monclient(hunting): get_auth_request con 0x7fc5fc1d6260 auth_method 0 2023-11-02T08:36:34.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7fc601211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7fc601211640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.761 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.714+0000 7fc601211640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.715+0000 7f9714bab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.715+0000 7f9714bab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.715+0000 7f9714bab640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.762 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.715+0000 7fc601211640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.715+0000 7fc601211640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.715+0000 7fc601211640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.763 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.715+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97100ada70 con 0x7f97100b0ed0 2023-11-02T08:36:34.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.722+0000 7fc05be84640 10 monclient: get_monmap_and_config 2023-11-02T08:36:34.764 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.722+0000 7fc05be84640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.765 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.722+0000 7fc05be84640 10 monclient: monmap: 2023-11-02T08:36:34.765 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.765 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.765 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:34.723318+0000 2023-11-02T08:36:34.765 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:34.723318+0000 2023-11-02T08:36:34.766 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.766 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.766 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:34.766 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:34.766 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:34.767 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:34.767 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:34.767 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.723+0000 7fc05be84640 1 Processor -- start 2023-11-02T08:36:34.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.723+0000 7fc05be84640 1 -- start start 2023-11-02T08:36:34.767 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.723+0000 7fc05be84640 10 monclient: init 2023-11-02T08:36:34.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.724+0000 7fc05be84640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.724+0000 7fc05be84640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-11-02T08:36:34.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.724+0000 7fc05be84640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0540b4570 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:34.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.724+0000 7fc05be84640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0540b0ed0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:34.768 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.724+0000 7fc05be84640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 0x7fc05413ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.724+0000 7fc05be84640 10 monclient(hunting): picked mon.noname-b con 0x7fc054144d40 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.724+0000 7fc05be84640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0540a9780 con 0x7fc0540b4570 2023-11-02T08:36:34.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.724+0000 7fc05be84640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0541451d0 con 0x7fc0540b0ed0 2023-11-02T08:36:34.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.724+0000 7fc05be84640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.769 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.725+0000 7fc05be84640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540aa1d0 con 0x7fc054144d40 2023-11-02T08:36:34.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.725+0000 7fc05be84640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.770 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.725+0000 7fc05be84640 10 monclient(hunting): authenticate will time out at 1366.080927s 2023-11-02T08:36:34.771 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.736+0000 7f03f4f18640 10 monclient: get_monmap_and_config 2023-11-02T08:36:34.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.736+0000 7f03f4f18640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.772 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.736+0000 7f03f4f18640 10 monclient: monmap: 2023-11-02T08:36:34.772 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.772 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.773 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:34.737704+0000 2023-11-02T08:36:34.773 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:34.737704+0000 2023-11-02T08:36:34.773 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.773 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.773 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:34.773 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:34.774 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:34.774 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:34.774 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:34.774 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.774 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.737+0000 7f03f4f18640 1 Processor -- start 2023-11-02T08:36:34.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.737+0000 7f03f4f18640 1 -- start start 2023-11-02T08:36:34.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.737+0000 7f03f4f18640 10 monclient: init 2023-11-02T08:36:34.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-11-02T08:36:34.775 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03f00b4570 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:34.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f03f00b0ed0 0x7f03f0144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 10 monclient(hunting): picked mon.noname-c con 0x7f03f00b0ed0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:34.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f03f013e520 0x7f03f013e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 10 monclient(hunting): picked mon.noname-a con 0x7f03f013e520 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:34.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f00a9780 con 0x7f03f00b4570 2023-11-02T08:36:34.776 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f00b4950 con 0x7f03f013e520 2023-11-02T08:36:34.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f00aa1d0 con 0x7f03f00b0ed0 2023-11-02T08:36:34.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.777 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.738+0000 7f03f4f18640 10 monclient(hunting): authenticate will time out at 1366.094826s 2023-11-02T08:36:34.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.742+0000 7f96feffd640 1 -- 172.21.15.151:0/4262689732 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f970003a040 con 0x7f97100b0ed0 2023-11-02T08:36:34.778 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.742+0000 7f96feffd640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f96f006be40 0x7f96f006e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.742+0000 7f970dd9b640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f96f006be40 0x7f96f006e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7f970dd9b640 10 monclient: get_auth_request con 0x7f96f006be40 auth_method 0 2023-11-02T08:36:34.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7f970dd9b640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f96f006be40 0x7f96f006e300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f97101d4220 tx=0x7f9704006c80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.779 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7f96feffd640 10 client.4378.objecter ms_handle_connect 0x7f96f006be40 2023-11-02T08:36:34.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2023-11-02T08:36:34.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 1 -- 172.21.15.151:0/3474826060 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5fc0ab560 msgr2=0x7fc5fc0ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5fc0ab560 0x7fc5fc0ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.780 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 1 -- 172.21.15.151:0/3474826060 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec005380 msgr2=0x7fc5fc0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec005380 0x7fc5fc0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 1 monclient: found mon.noname-b 2023-11-02T08:36:34.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.781 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.782 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.743+0000 7fc601211640 1 -- 172.21.15.151:0/3474826060 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc0aaf50 con 0x7fc5fc1d6260 2023-11-02T08:36:34.782 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.764+0000 7fdea57fa640 1 -- 172.21.15.151:0/2086077792 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdea003e920 con 0x7fdea813e520 2023-11-02T08:36:34.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.764+0000 7fdea57fa640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fde8c06bdc0 0x7fde8c06e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.764+0000 7fdeacbaa640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fde8c06bdc0 0x7fde8c06e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.764+0000 7fdeacbaa640 10 monclient: get_auth_request con 0x7fde8c06bdc0 auth_method 0 2023-11-02T08:36:34.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.765+0000 7fdea57fa640 10 client.4289.objecter ms_handle_connect 0x7fde8c06bdc0 2023-11-02T08:36:34.783 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.765+0000 7fdeacbaa640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fde8c06bdc0 0x7fde8c06e280 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde9c001560 tx=0x7fde9c007990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.768+0000 7fdea57fa640 1 -- 172.21.15.151:0/2086077792 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdea007b6b0 con 0x7fdea813e520 2023-11-02T08:36:34.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.768+0000 7fdea57fa640 10 client.4289.objecter ms_dispatch 0x7fdea8000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:34.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.768+0000 7fdea57fa640 3 client.4289.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:34.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.768+0000 7fdea57fa640 3 client.4289.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:34.784 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.768+0000 7fdea57fa640 20 client.4289.objecter dump_active .. 0 homeless 2023-11-02T08:36:34.785 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.780+0000 7fc05a421640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 0x7fc05413ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.788 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.785+0000 7f03eed9d640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f03f013e520 0x7f03f013e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.795+0000 7fc05a421640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 0x7fc05413ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:54998/0 (socket says 172.21.15.151:54998) 2023-11-02T08:36:34.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.795+0000 7fc05a421640 1 -- 172.21.15.151:0/1539630096 learned_addr learned my addr 172.21.15.151:0/1539630096 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.798 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.795+0000 7fc05a421640 10 monclient(hunting): get_auth_request con 0x7fc054144d40 auth_method 0 2023-11-02T08:36:34.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.795+0000 7fc05a421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.795+0000 7fc05a421640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.795+0000 7fc05a421640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.799 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.795+0000 7fdea57fa640 1 -- 172.21.15.151:0/2086077792 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fdea00772f0 con 0x7fdea813e520 2023-11-02T08:36:34.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.795+0000 7fdea57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.800 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.795+0000 7fdea57fa640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:34.813 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.810+0000 7f03ee59c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f03f00b4570 0x7f03f00a9070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:34186/0 (socket says 172.21.15.151:34186) 2023-11-02T08:36:34.814 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.810+0000 7f03ee59c640 1 -- 172.21.15.151:0/4270658489 learned_addr learned my addr 172.21.15.151:0/4270658489 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.814+0000 7f03eed9d640 10 monclient(hunting): get_auth_request con 0x7f03f013e520 auth_method 0 2023-11-02T08:36:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.814+0000 7f03eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.814+0000 7f03eed9d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.817 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.814+0000 7f03eed9d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.816+0000 7f3f7ff49640 10 monclient: get_monmap_and_config 2023-11-02T08:36:34.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.816+0000 7f3f7ff49640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.820 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.816+0000 7f3f7ff49640 10 monclient: monmap: 2023-11-02T08:36:34.820 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.821 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.821 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:34.818052+0000 2023-11-02T08:36:34.821 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:34.818052+0000 2023-11-02T08:36:34.821 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.821 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.822 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:34.822 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:34.822 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:34.822 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:34.822 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:34.823 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.824 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.817+0000 7f3f7ff49640 1 Processor -- start 2023-11-02T08:36:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.817+0000 7f3f7ff49640 1 -- start start 2023-11-02T08:36:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.817+0000 7f3f7ff49640 10 monclient: init 2023-11-02T08:36:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.825 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-11-02T08:36:34.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3f780b4570 0x7f3f78144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient(hunting): picked mon.noname-a con 0x7f3f780b4570 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:34.826 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3f780b0ed0 0x7f3f780a9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient(hunting): picked mon.noname-c con 0x7f3f780b0ed0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:34.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b1cb0 0x7f3f780b2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient(hunting): picked mon.noname-b con 0x7f3f780b1cb0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.827 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f780b4950 con 0x7f3f780b4570 2023-11-02T08:36:34.828 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.828 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f780aa1d0 con 0x7f3f780b0ed0 2023-11-02T08:36:34.828 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.828 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780aa4b0 con 0x7f3f780b1cb0 2023-11-02T08:36:34.828 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.828 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.818+0000 7f3f7ff49640 10 monclient(hunting): authenticate will time out at 1366.174776s 2023-11-02T08:36:34.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.820+0000 7f03eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.820+0000 7f03eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.829 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.820+0000 7f03eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.821+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982974114 0 0) 0x7fc0540a9780 con 0x7fc0540b4570 2023-11-02T08:36:34.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.821+0000 7fc058c1e640 10 monclient(hunting): _negotiate 2023-11-02T08:36:34.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.821+0000 7fc058c1e640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.830 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.821+0000 7fc058c1e640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.821+0000 7fc058c1e640 10 monclient(hunting): my global_id is 4326 2023-11-02T08:36:34.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.821+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc040003520 con 0x7fc0540b4570 2023-11-02T08:36:34.831 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.822+0000 7f3f7dce5640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3f780b4570 0x7f3f78144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.824+0000 7f03eed9d640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2023-11-02T08:36:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.824+0000 7f03eed9d640 1 -- 172.21.15.151:0/4270658489 >> v2:172.21.15.178:3300/0 conn(0x7f03f00b0ed0 msgr2=0x7f03f0144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.824+0000 7f03eed9d640 1 --2- 172.21.15.151:0/4270658489 >> v2:172.21.15.178:3300/0 conn(0x7f03f00b0ed0 0x7f03f0144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.833 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.824+0000 7f03eed9d640 1 -- 172.21.15.151:0/4270658489 >> v1:172.21.15.151:6789/0 conn(0x7f03f00b4570 legacy=0x7f03f00a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.833 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.824+0000 7f03eed9d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.833 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.824+0000 7f03eed9d640 1 monclient: found mon.noname-a 2023-11-02T08:36:34.833 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.824+0000 7f03eed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.824+0000 7f03eed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.835 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.824+0000 7f03eed9d640 1 -- 172.21.15.151:0/4270658489 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f00aa4b0 con 0x7f03f013e520 2023-11-02T08:36:34.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.825+0000 7f3f7dce5640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3f780b4570 0x7f3f78144d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38800/0 (socket says 172.21.15.151:38800) 2023-11-02T08:36:34.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.825+0000 7f3f7dce5640 1 -- 172.21.15.151:0/2876094998 learned_addr learned my addr 172.21.15.151:0/2876094998 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.836 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.825+0000 7f3f7dce5640 10 monclient(hunting): get_auth_request con 0x7f3f780b4570 auth_method 0 2023-11-02T08:36:34.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.825+0000 7f3f7dce5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.825+0000 7f3f7dce5640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.825+0000 7f3f7dce5640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.837 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.834+0000 7f3f7dce5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.838 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.834+0000 7f3f7dce5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.839 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.834+0000 7f3f7dce5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.841 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2023-11-02T08:36:34.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 1 -- 172.21.15.151:0/2876094998 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b1cb0 msgr2=0x7f3f780b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 1 --2- 172.21.15.151:0/2876094998 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b1cb0 0x7f3f780b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 1 -- 172.21.15.151:0/2876094998 >> v2:172.21.15.178:3300/0 conn(0x7f3f780b0ed0 msgr2=0x7f3f780a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 1 --2- 172.21.15.151:0/2876094998 >> v2:172.21.15.178:3300/0 conn(0x7f3f780b0ed0 0x7f3f780a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.842 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 1 monclient: found mon.noname-a 2023-11-02T08:36:34.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.843 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.844 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.838+0000 7f3f7dce5640 1 -- 172.21.15.151:0/2876094998 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f780a8b50 con 0x7f3f780b4570 2023-11-02T08:36:34.847 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.844+0000 7f03eed9d640 1 --2- 172.21.15.151:0/4270658489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f013e520 0x7f03f013e9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009d90 tx=0x7f03e802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837c832d3920446c server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.853 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.850+0000 7f3f7dce5640 1 --2- 172.21.15.151:0/2876094998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f780b4570 0x7f3f78144d40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f74009870 tx=0x7f3f7402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344986b83d3213c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4195726609 0 0) 0x7fc040003520 con 0x7fc0540b4570 2023-11-02T08:36:34.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 msgr2=0x7fc05413ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 1 --2- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 0x7fc05413ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 >> v1:172.21.15.178:6789/0 conn(0x7fc0540b0ed0 legacy=0x7fc05413e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 1 monclient: found mon.noname-a-legacy 2023-11-02T08:36:34.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.151:6789/0 2023-11-02T08:36:34.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0540aa4b0 con 0x7fc0540b4570 2023-11-02T08:36:34.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc05be84640 5 monclient: authenticate success, global_id 4326 2023-11-02T08:36:34.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc05be84640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:34.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 650167734 0 0) 0x7fc050001650 con 0x7fc0540b4570 2023-11-02T08:36:34.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-11-02T08:36:34.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 10 monclient: dump: 2023-11-02T08:36:34.869 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.869 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.870 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.870 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.870 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.870 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.870 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.871 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.871 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.871 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] but i'm connected to v1:172.21.15.151:6789/0, reconnecting 2023-11-02T08:36:34.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 >> v1:172.21.15.151:6789/0 conn(0x7fc0540b4570 legacy=0x7fc0540a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.861+0000 7fc058c1e640 10 monclient: _add_conns ranks=[1,2,0] 2023-11-02T08:36:34.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 1 --2- 172.21.15.151:0/1539630096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0400039c0 0x7fc040003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 10 monclient(hunting): picked mon.b con 0x7fc0400039c0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 1 --2- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 0x7fc0400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 10 monclient(hunting): picked mon.c con 0x7fc054144d40 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 1 --2- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc040004d50 0x7fc0400051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 10 monclient(hunting): picked mon.a con 0x7fc040004d50 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540aa1d0 con 0x7fc040004d50 2023-11-02T08:36:34.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04000bbe0 con 0x7fc0400039c0 2023-11-02T08:36:34.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04000bda0 con 0x7fc054144d40 2023-11-02T08:36:34.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc058c1e640 10 monclient(hunting): _finish_auth 0 2023-11-02T08:36:34.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.862+0000 7fc05be84640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:34.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.865+0000 7fc05941f640 1 --2- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 0x7fc0400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.865+0000 7fc05941f640 10 monclient(hunting): get_auth_request con 0x7fc054144d40 auth_method 0 2023-11-02T08:36:34.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.865+0000 7fc05941f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.865+0000 7fc05941f640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.865+0000 7fc05941f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-11-02T08:36:34.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.867+0000 7fc05941f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.867+0000 7fc05941f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.867+0000 7fc05941f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-11-02T08:36:34.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 10 monclient(hunting): handle_auth_done global_id 4326 payload 293 2023-11-02T08:36:34.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 1 -- 172.21.15.151:0/1539630096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0400039c0 msgr2=0x7fc040003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 1 --2- 172.21.15.151:0/1539630096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0400039c0 0x7fc040003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 1 -- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc040004d50 msgr2=0x7fc0400051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:34.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 1 --2- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc040004d50 0x7fc0400051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 1 monclient: found mon.c 2023-11-02T08:36:34.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 1 -- 172.21.15.151:0/1539630096 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0540aa4b0 con 0x7fc054144d40 2023-11-02T08:36:34.880 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.870+0000 7fc05941f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.873+0000 7fc05941f640 1 --2- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 0x7fc0400047c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc044001630 tx=0x7fc0400057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6549ed959bf3dddb server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.881 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.877+0000 7faf35ffb640 1 -- 172.21.15.151:0/3515246389 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7faf3c076390 con 0x7faf40069ee0 2023-11-02T08:36:34.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.877+0000 7faf35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.877+0000 7faf35ffb640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:34.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044005270 con 0x7fc054144d40 2023-11-02T08:36:34.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:34.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 10 monclient: dump: 2023-11-02T08:36:34.884 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.884 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.884 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.884 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.885 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.885 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.885 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.885 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.885 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.886 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04402e070 con 0x7fc054144d40 2023-11-02T08:36:34.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 1 -- 172.21.15.151:0/1539630096 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044003610 con 0x7fc054144d40 2023-11-02T08:36:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc058c1e640 10 monclient: dump: 2023-11-02T08:36:34.887 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.888 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.888 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.888 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.888 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.888 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.888 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.889 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.889 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.889 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc05be84640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:34.889 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.878+0000 7fc05be84640 4 set_mon_vals no callback set 2023-11-02T08:36:34.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.879+0000 7fc05be84640 10 monclient: shutdown 2023-11-02T08:36:34.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.879+0000 7fc05be84640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:34.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.880+0000 7fc05be84640 1 -- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 msgr2=0x7fc0400047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.890 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.880+0000 7fc05be84640 1 --2- 172.21.15.151:0/1539630096 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc054144d40 0x7fc0400047c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc044001630 tx=0x7fc0400057f0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.880+0000 7fc05a421640 1 -- 172.21.15.151:0/1539630096 reap_dead start 2023-11-02T08:36:34.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.880+0000 7fc05be84640 1 -- 172.21.15.151:0/1539630096 shutdown_connections 2023-11-02T08:36:34.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.880+0000 7fc05be84640 1 -- 172.21.15.151:0/1539630096 >> 172.21.15.151:0/1539630096 conn(0x7fc054065d50 msgr2=0x7fc054066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:34.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.880+0000 7fc05be84640 1 -- 172.21.15.151:0/1539630096 shutdown_connections 2023-11-02T08:36:34.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.880+0000 7fc05be84640 1 -- 172.21.15.151:0/1539630096 wait complete. 2023-11-02T08:36:34.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.881+0000 7fc05be84640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.881+0000 7fc05be84640 10 monclient: monmap: 2023-11-02T08:36:34.894 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.895 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:34.895 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:34.895 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:34.895 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.895 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.896 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:34.896 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:34.896 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:34.896 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.882+0000 7fc05be84640 1 librados: starting msgr at 2023-11-02T08:36:34.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.882+0000 7fc05be84640 1 librados: starting objecter 2023-11-02T08:36:34.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.882+0000 7fc05be84640 1 Processor -- start 2023-11-02T08:36:34.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.883+0000 7fc05be84640 1 -- start start 2023-11-02T08:36:34.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.883+0000 7fc05be84640 1 librados: setting wanted keys 2023-11-02T08:36:34.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.883+0000 7fc05be84640 1 librados: calling monclient init 2023-11-02T08:36:34.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.883+0000 7fc05be84640 10 monclient: init 2023-11-02T08:36:34.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.884+0000 7fc05be84640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.884+0000 7fc05be84640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:36:34.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.884+0000 7fc05be84640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0541d9c00 0x7fc054136530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.884+0000 7fc05be84640 10 monclient(hunting): picked mon.noname-a con 0x7fc0541d9c00 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.884+0000 7fc05be84640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0541da620 0x7fc054136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.884+0000 7fc05be84640 10 monclient(hunting): picked mon.noname-c con 0x7fc0541da620 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc054143590 0x7fc054143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 10 monclient(hunting): picked mon.noname-b con 0x7fc054143590 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0541d9fe0 con 0x7fc0541d9c00 2023-11-02T08:36:34.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc054136fb0 con 0x7fc054143590 2023-11-02T08:36:34.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc054142dd0 con 0x7fc0541da620 2023-11-02T08:36:34.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.885+0000 7fc05be84640 10 monclient(hunting): authenticate will time out at 1366.241589s 2023-11-02T08:36:34.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.886+0000 7f3f7cce3640 1 -- 172.21.15.151:0/2876094998 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f74031740 con 0x7f3f780b4570 2023-11-02T08:36:34.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.886+0000 7f3f7cce3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:34.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 10 monclient: dump: 2023-11-02T08:36:34.904 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.904 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.904 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.904 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.905 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.905 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.905 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.905 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.905 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.906 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 1 -- 172.21.15.151:0/2876094998 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f740318a0 con 0x7f3f780b4570 2023-11-02T08:36:34.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7ff49640 5 monclient: authenticate success, global_id 4332 2023-11-02T08:36:34.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7ff49640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:34.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 1 -- 172.21.15.151:0/2876094998 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f74031c30 con 0x7f3f780b4570 2023-11-02T08:36:34.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:34.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7cce3640 10 monclient: dump: 2023-11-02T08:36:34.908 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.908 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.909 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.909 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.909 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.909 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.910 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7ff49640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:34.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.887+0000 7f3f7ff49640 4 set_mon_vals no callback set 2023-11-02T08:36:34.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 10 monclient: shutdown 2023-11-02T08:36:34.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:34.911 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2876094998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f780b4570 msgr2=0x7f3f78144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2876094998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f780b4570 0x7f3f78144d40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f74009870 tx=0x7f3f7402b7e0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2876094998 shutdown_connections 2023-11-02T08:36:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2876094998 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b1cb0 0x7f3f780b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2876094998 >> v2:172.21.15.178:3300/0 conn(0x7f3f780b0ed0 0x7f3f780a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.912 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2876094998 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f780b4570 0x7f3f78144d40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2876094998 >> 172.21.15.151:0/2876094998 conn(0x7f3f78065d50 msgr2=0x7f3f78066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2876094998 shutdown_connections 2023-11-02T08:36:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.888+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2876094998 wait complete. 2023-11-02T08:36:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f3f7ff49640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f3f7ff49640 10 monclient: monmap: 2023-11-02T08:36:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:34.914 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:34.915 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:34.915 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 1 -- 172.21.15.151:0/4270658489 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e802cb40 con 0x7f03f013e520 2023-11-02T08:36:34.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:34.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 10 monclient: dump: 2023-11-02T08:36:34.917 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.918 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.919 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.919 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 1 -- 172.21.15.151:0/4270658489 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e802cca0 con 0x7f03f013e520 2023-11-02T08:36:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03f4f18640 5 monclient: authenticate success, global_id 4329 2023-11-02T08:36:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03f4f18640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.922 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 1 -- 172.21.15.151:0/4270658489 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8034660 con 0x7f03f013e520 2023-11-02T08:36:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03ed59a640 10 monclient: dump: 2023-11-02T08:36:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.924 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.925 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.925 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03f4f18640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:34.926 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f03f4f18640 4 set_mon_vals no callback set 2023-11-02T08:36:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f3f7ff49640 1 librados: starting msgr at 2023-11-02T08:36:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.889+0000 7f3f7ff49640 1 librados: starting objecter 2023-11-02T08:36:34.927 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.890+0000 7f3f7ff49640 1 Processor -- start 2023-11-02T08:36:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.890+0000 7f3f7ff49640 1 -- start start 2023-11-02T08:36:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.890+0000 7f3f7ff49640 1 librados: setting wanted keys 2023-11-02T08:36:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.890+0000 7f3f7ff49640 1 librados: calling monclient init 2023-11-02T08:36:34.928 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.890+0000 7f3f7ff49640 10 monclient: init 2023-11-02T08:36:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.890+0000 7f03f4f18640 10 monclient: shutdown 2023-11-02T08:36:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.890+0000 7f03f4f18640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:34.929 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.890+0000 7f03f4f18640 1 -- 172.21.15.151:0/4270658489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f013e520 msgr2=0x7f03f013e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.890+0000 7f03f4f18640 1 --2- 172.21.15.151:0/4270658489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f013e520 0x7f03f013e9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009d90 tx=0x7f03e802bd90 comp rx=0 tx=0).stop 2023-11-02T08:36:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.891+0000 7f03f4f18640 1 -- 172.21.15.151:0/4270658489 shutdown_connections 2023-11-02T08:36:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.891+0000 7f03f4f18640 1 --2- 172.21.15.151:0/4270658489 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f013e520 0x7f03f013e9b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.891+0000 7f03f4f18640 1 --2- 172.21.15.151:0/4270658489 >> v2:172.21.15.178:3300/0 conn(0x7f03f00b0ed0 0x7f03f0144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.930 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.891+0000 7f03f4f18640 1 -- 172.21.15.151:0/4270658489 >> 172.21.15.151:0/4270658489 conn(0x7f03f0065d50 msgr2=0x7f03f0066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.891+0000 7f03f4f18640 1 -- 172.21.15.151:0/4270658489 shutdown_connections 2023-11-02T08:36:34.931 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.891+0000 7f03f4f18640 1 -- 172.21.15.151:0/4270658489 wait complete. 2023-11-02T08:36:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.891+0000 7f3f7ff49640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.891+0000 7f3f7ff49640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:36:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.891+0000 7f3f7ff49640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f780b0ed0 0x7f3f780ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 10 monclient(hunting): picked mon.noname-a con 0x7f3f780b0ed0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f780b1cb0 0x7f3f781da530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 10 monclient(hunting): picked mon.noname-b con 0x7f3f780b1cb0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b4570 0x7f3f781daa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 10 monclient(hunting): picked mon.noname-c con 0x7f3f780b4570 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.933 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f781dafb0 con 0x7f3f780b0ed0 2023-11-02T08:36:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780ad730 con 0x7f3f780b1cb0 2023-11-02T08:36:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.934 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f781da010 con 0x7f3f780b4570 2023-11-02T08:36:34.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.935 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f3f7ff49640 10 monclient(hunting): authenticate will time out at 1366.248394s 2023-11-02T08:36:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.892+0000 7f03f4f18640 10 monclient: build_initial_monmap 2023-11-02T08:36:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.893+0000 7f03f4f18640 10 monclient: monmap: 2023-11-02T08:36:34.936 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:34.937 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:34.937 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:34.937 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:34.937 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:34.937 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.938 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:34.938 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:34.938 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:34.938 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.938 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.893+0000 7f03f4f18640 1 librados: starting msgr at 2023-11-02T08:36:34.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.893+0000 7f03f4f18640 1 librados: starting objecter 2023-11-02T08:36:34.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7f03f4f18640 1 Processor -- start 2023-11-02T08:36:34.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7f03f4f18640 1 -- start start 2023-11-02T08:36:34.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7f03f4f18640 1 librados: setting wanted keys 2023-11-02T08:36:34.939 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7f03f4f18640 1 librados: calling monclient init 2023-11-02T08:36:34.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7f03f4f18640 10 monclient: init 2023-11-02T08:36:34.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7fc05941f640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0541da620 0x7fc054136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7fc05941f640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0541da620 0x7fc054136a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:55026/0 (socket says 172.21.15.151:55026) 2023-11-02T08:36:34.940 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7fc05941f640 1 -- 172.21.15.151:0/3183502727 learned_addr learned my addr 172.21.15.151:0/3183502727 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7fc05941f640 10 monclient(hunting): get_auth_request con 0x7fc0541da620 auth_method 0 2023-11-02T08:36:34.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7fc05941f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7fc05941f640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.941 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.894+0000 7fc05941f640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.895+0000 7f3f7e4e6640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b4570 0x7f3f781daa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7fc05941f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7fc05941f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.942 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7fc05941f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:34.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:36:34.944 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f00b0ed0 0x7f03f0141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient(hunting): picked mon.noname-a con 0x7f03f00b0ed0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:34.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f03f00b4570 0x7f03f00ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient(hunting): picked mon.noname-c con 0x7f03f00b4570 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:34.945 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03f013e520 0x7f03f00acb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:34.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient(hunting): picked mon.noname-b con 0x7f03f013e520 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:34.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f00ad2a0 con 0x7f03f00b0ed0 2023-11-02T08:36:34.946 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f0070fb0 con 0x7f03f013e520 2023-11-02T08:36:34.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:34.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f0071120 con 0x7f03f00b4570 2023-11-02T08:36:34.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:34.947 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.896+0000 7f03f4f18640 10 monclient(hunting): authenticate will time out at 1366.252102s 2023-11-02T08:36:34.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.897+0000 7fc059c20640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0541d9c00 0x7fc054136530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.948 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.899+0000 7fc059c20640 10 monclient(hunting): get_auth_request con 0x7fc0541d9c00 auth_method 0 2023-11-02T08:36:34.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.899+0000 7fc059c20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.899+0000 7fc059c20640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.899+0000 7fc059c20640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.949 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc059c20640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc059c20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc059c20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2023-11-02T08:36:34.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 1 -- 172.21.15.151:0/3183502727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc054143590 msgr2=0x7fc054143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.950 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc054143590 0x7fc054143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 1 -- 172.21.15.151:0/3183502727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0541d9c00 msgr2=0x7fc054136530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0541d9c00 0x7fc054136530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 1 monclient: found mon.noname-c 2023-11-02T08:36:34.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.951 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.952 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.901+0000 7fc05941f640 1 -- 172.21.15.151:0/3183502727 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc054143070 con 0x7fc0541da620 2023-11-02T08:36:34.952 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.902+0000 7f3f7e4e6640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b4570 0x7f3f781daa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:55036/0 (socket says 172.21.15.151:55036) 2023-11-02T08:36:34.952 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.902+0000 7f3f7e4e6640 1 -- 172.21.15.151:0/2921646097 learned_addr learned my addr 172.21.15.151:0/2921646097 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.902+0000 7f3f7e4e6640 10 monclient(hunting): get_auth_request con 0x7f3f780b4570 auth_method 0 2023-11-02T08:36:34.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.902+0000 7f3f7e4e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.902+0000 7f3f7e4e6640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.902+0000 7f3f7e4e6640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.953 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.905+0000 7f3f7e4e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.905+0000 7f3f7e4e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.905+0000 7f3f7e4e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.907+0000 7fc05941f640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0541da620 0x7fc054136a70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc04402bbc0 tx=0x7fc044001000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.954 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.907+0000 7fc04a7fc640 10 client.?.objecter ms_handle_connect 0x7fc0541da620 2023-11-02T08:36:34.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.907+0000 7fc04a7fc640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:34.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.909+0000 7f85f2c1c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-02T08:36:34.955 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.909+0000 7f85f2c1c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-02T08:36:34.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.909+0000 7f85f2c1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:34.956 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.909+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f85ec1d9800 con 0x7f85ec0b0ed0 2023-11-02T08:36:34.957 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2023-11-02T08:36:34.957 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 1 -- 172.21.15.151:0/2921646097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f780b1cb0 msgr2=0x7f3f781da530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.957 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f780b1cb0 0x7f3f781da530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.957 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 1 -- 172.21.15.151:0/2921646097 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f780b0ed0 msgr2=0x7f3f780ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.957 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f780b0ed0 0x7f3f780ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 1 monclient: found mon.noname-c 2023-11-02T08:36:34.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.958 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.912+0000 7f3f7e4e6640 1 -- 172.21.15.151:0/2921646097 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f740094d0 con 0x7f3f780b4570 2023-11-02T08:36:34.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.914+0000 7f03ee59c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f00b0ed0 0x7f03f0141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:34.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.914+0000 7f03ee59c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f00b0ed0 0x7f03f0141510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38832/0 (socket says 172.21.15.151:38832) 2023-11-02T08:36:34.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.914+0000 7f03ee59c640 1 -- 172.21.15.151:0/3416824893 learned_addr learned my addr 172.21.15.151:0/3416824893 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:34.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.915+0000 7f03ee59c640 10 monclient(hunting): get_auth_request con 0x7f03f00b0ed0 auth_method 0 2023-11-02T08:36:34.960 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.915+0000 7f03ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:34.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.915+0000 7f03ee59c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:34.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.915+0000 7f03ee59c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:34.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.916+0000 7f03ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:34.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.916+0000 7f03ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:34.961 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.916+0000 7f03ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:34.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.918+0000 7f3f7e4e6640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b4570 0x7f3f781daa70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000b970 tx=0x7f3f7000be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.918+0000 7f3f6e7fc640 10 client.?.objecter ms_handle_connect 0x7f3f780b4570 2023-11-02T08:36:34.962 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.918+0000 7f3f6e7fc640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:34.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 1 -- 172.21.15.151:0/3183502727 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044003a80 con 0x7fc0541da620 2023-11-02T08:36:34.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.963 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:34.964 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 10 monclient: dump: 2023-11-02T08:36:34.964 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.964 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.964 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.964 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.965 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.965 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.965 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.965 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.965 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.966 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.966 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 1 -- 172.21.15.151:0/3183502727 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc044003be0 con 0x7fc0541da620 2023-11-02T08:36:34.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 1 -- 172.21.15.151:0/3183502727 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044033430 con 0x7fc0541da620 2023-11-02T08:36:34.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc05be84640 5 monclient: authenticate success, global_id 4295 2023-11-02T08:36:34.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:34.967 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04a7fc640 10 monclient: dump: 2023-11-02T08:36:34.968 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.968 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.968 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.968 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.968 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.969 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.969 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.969 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.969 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.969 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc04bfff640 4 set_mon_vals no callback set 2023-11-02T08:36:34.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc05be84640 10 monclient: _renew_subs 2023-11-02T08:36:34.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc05be84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0541d3f50 con 0x7fc0541da620 2023-11-02T08:36:34.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc05be84640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:34.970 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc05be84640 10 monclient: _renew_subs 2023-11-02T08:36:34.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc05be84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc054143f50 con 0x7fc0541da620 2023-11-02T08:36:34.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.919+0000 7fc05be84640 1 librados: init done 2023-11-02T08:36:34.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7fc05be84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.971 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7fc05be84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7fc05be84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:34.972 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0540ab4d0 con 0x7fc0541da620 2023-11-02T08:36:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2023-11-02T08:36:34.973 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 1 -- 172.21.15.151:0/3416824893 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f03f00b4570 msgr2=0x7f03f00ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f03f00b4570 0x7f03f00ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 1 -- 172.21.15.151:0/3416824893 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03f013e520 msgr2=0x7f03f00acb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:34.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03f013e520 0x7f03f00acb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:34.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 10 monclient: _finish_hunting 0 2023-11-02T08:36:34.974 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 1 monclient: found mon.noname-a 2023-11-02T08:36:34.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:34.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.921+0000 7f03ee59c640 1 -- 172.21.15.151:0/3416824893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e8009a70 con 0x7f03f00b0ed0 2023-11-02T08:36:34.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.928+0000 7f03ee59c640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f00b0ed0 0x7f03f0141510 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03e000c4a0 tx=0x7f03e000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:34.975 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.928+0000 7f03deffd640 10 client.?.objecter ms_handle_connect 0x7f03f00b0ed0 2023-11-02T08:36:34.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.928+0000 7f03deffd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:34.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 1 -- 172.21.15.151:0/3416824893 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0005ce0 con 0x7f03f00b0ed0 2023-11-02T08:36:34.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:34.976 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 10 monclient: dump: 2023-11-02T08:36:34.977 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.977 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.977 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.977 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.977 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.978 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.978 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.978 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.978 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.978 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.978 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 10 monclient: _finish_auth 0 2023-11-02T08:36:34.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:34.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:34.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 1 -- 172.21.15.151:0/3416824893 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e0005e40 con 0x7f03f00b0ed0 2023-11-02T08:36:34.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03f4f18640 5 monclient: authenticate success, global_id 4338 2023-11-02T08:36:34.979 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03f4f18640 10 monclient: _renew_subs 2023-11-02T08:36:34.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03f4f18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f0139690 con 0x7f03f00b0ed0 2023-11-02T08:36:34.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:34.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 1 -- 172.21.15.151:0/3416824893 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0007270 con 0x7f03f00b0ed0 2023-11-02T08:36:34.980 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:34.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:34.981 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03deffd640 10 monclient: dump: 2023-11-02T08:36:34.981 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:34.981 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:34.981 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.981 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:34.982 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:34.982 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:34.982 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:34.982 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:34.982 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:34.983 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:34.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03dffff640 4 set_mon_vals no callback set 2023-11-02T08:36:34.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03f4f18640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:34.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03f4f18640 10 monclient: _renew_subs 2023-11-02T08:36:34.983 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03f4f18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f0139bf0 con 0x7f03f00b0ed0 2023-11-02T08:36:34.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.936+0000 7f03f4f18640 1 librados: init done 2023-11-02T08:36:34.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.937+0000 7f03f4f18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.937+0000 7f03f4f18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:34.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.938+0000 7f03f4f18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:34.984 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:34.938+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f0067920 con 0x7f03f00b0ed0 2023-11-02T08:36:35.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.014+0000 7fdeaee0e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-02T08:36:35.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.014+0000 7fdeaee0e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-02T08:36:35.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.014+0000 7fdeaee0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:35.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.014+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdea80a8bb0 con 0x7fdea813e520 2023-11-02T08:36:35.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.031+0000 7f96feffd640 1 -- 172.21.15.151:0/4262689732 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f970007a730 con 0x7f97100b0ed0 2023-11-02T08:36:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.031+0000 7f96feffd640 10 client.4378.objecter ms_dispatch 0x7f9710000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.031+0000 7f96feffd640 3 client.4378.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.031+0000 7f96feffd640 3 client.4378.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.031+0000 7f96feffd640 20 client.4378.objecter dump_active .. 0 homeless 2023-11-02T08:36:35.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.044+0000 7f96feffd640 1 -- 172.21.15.151:0/4262689732 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f9700076370 con 0x7f97100b0ed0 2023-11-02T08:36:35.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.044+0000 7f96feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.044+0000 7f96feffd640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:35.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.050+0000 7fc601211640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5fc1d6260 0x7fc5fc1d66e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc5f402c800 tx=0x7fc5f4004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:35.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.050+0000 7fc5f97fa640 10 client.?.objecter ms_handle_connect 0x7fc5fc1d6260 2023-11-02T08:36:35.053 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.050+0000 7fc5f97fa640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:35.080 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 1 -- 172.21.15.151:0/3474826060 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4035c60 con 0x7fc5fc1d6260 2023-11-02T08:36:35.080 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:35.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:35.081 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 10 monclient: dump: 2023-11-02T08:36:35.081 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:35.081 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:35.081 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.082 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.082 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:35.082 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.082 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:35.082 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:35.083 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:35.083 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.083 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 10 monclient: _finish_auth 0 2023-11-02T08:36:35.083 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 10 monclient: _check_auth_tickets 2023-11-02T08:36:35.083 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:35.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 1 -- 172.21.15.151:0/3474826060 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f4035dc0 con 0x7fc5fc1d6260 2023-11-02T08:36:35.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:35.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 1 -- 172.21.15.151:0/3474826060 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f403d640 con 0x7fc5fc1d6260 2023-11-02T08:36:35.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:35.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:35.084 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5f97fa640 10 monclient: dump: 2023-11-02T08:36:35.085 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:35.085 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:35.085 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.085 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.085 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:35.086 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.086 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:35.086 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:35.086 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:35.086 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc602c74640 5 monclient: authenticate success, global_id 4381 2023-11-02T08:36:35.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc5faffd640 4 set_mon_vals no callback set 2023-11-02T08:36:35.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc602c74640 10 monclient: _renew_subs 2023-11-02T08:36:35.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc602c74640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:35.087 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5fc0ab140 con 0x7fc5fc1d6260 2023-11-02T08:36:35.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc602c74640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:35.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc602c74640 10 monclient: _renew_subs 2023-11-02T08:36:35.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc602c74640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:35.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5fc13a650 con 0x7fc5fc1d6260 2023-11-02T08:36:35.088 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.076+0000 7fc602c74640 1 librados: init done 2023-11-02T08:36:35.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.078+0000 7fc602c74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.089 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.078+0000 7fc602c74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.078+0000 7fc602c74640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:35.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.078+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5fc0aac70 con 0x7fc5fc1d6260 2023-11-02T08:36:35.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.078+0000 7f85e97fa640 1 -- 172.21.15.151:0/1120937566 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f85ec1d9800 con 0x7f85ec0b0ed0 2023-11-02T08:36:35.090 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.079+0000 7f85e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-11-02T08:36:35.091 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.079+0000 7f85e97fa640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:35.091 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:35.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.079+0000 7faf476ca640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-02T08:36:35.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.079+0000 7faf476ca640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-02T08:36:35.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.079+0000 7faf476ca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:35.092 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.079+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faf400b2380 con 0x7faf40069ee0 2023-11-02T08:36:35.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 10 librados: watch_flush enter 2023-11-02T08:36:35.093 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 10 librados: watch_flush exit 2023-11-02T08:36:35.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 20 client.4369.objecter shutdown clearing up homeless session... 2023-11-02T08:36:35.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 10 client.4369.objecter successfully canceled tick 2023-11-02T08:36:35.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f85d006be10 msgr2=0x7f85d006e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.094 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f85d006be10 0x7f85d006e2d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f85d8036000 tx=0x7f85d8030000 comp rx=0 tx=0).stop 2023-11-02T08:36:35.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 10 monclient: shutdown 2023-11-02T08:36:35.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:35.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 msgr2=0x7f85ec1d9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.081+0000 7f85f2c1c640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 0x7f85ec1d9e80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f85e000d5f0 tx=0x7f85e000dab0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.095 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.082+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 shutdown_connections 2023-11-02T08:36:35.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.082+0000 7f85f2c1c640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f85ec0b4570 0x7f85ec0acc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.082+0000 7f85f2c1c640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f85ec0b1cb0 0x7f85ec0ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.082+0000 7f85f2c1c640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f85d006be10 0x7f85d006e2d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.082+0000 7f85f2c1c640 1 --2- 172.21.15.151:0/1120937566 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85ec0b0ed0 0x7f85ec1d9e80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.096 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.082+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 >> 172.21.15.151:0/1120937566 conn(0x7f85ec065d50 msgr2=0x7f85ec13ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:35.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.082+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 shutdown_connections 2023-11-02T08:36:35.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.082+0000 7f85f2c1c640 1 -- 172.21.15.151:0/1120937566 wait complete. 2023-11-02T08:36:35.097 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.082+0000 7f85f2c1c640 1 librados: shutdown 2023-11-02T08:36:35.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.6 2023-11-02T08:36:35.100 DEBUG:teuthology.parallel:result is None 2023-11-02T08:36:35.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.197+0000 7fc04a7fc640 1 -- 172.21.15.151:0/3183502727 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc044030030 con 0x7fc0541da620 2023-11-02T08:36:35.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.197+0000 7fc04a7fc640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc03806bdf0 0x7fc03806e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:35.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.197+0000 7fc059c20640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc03806bdf0 0x7fc03806e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:35.201 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.198+0000 7fc059c20640 10 monclient: get_auth_request con 0x7fc03806bdf0 auth_method 0 2023-11-02T08:36:35.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.198+0000 7fc059c20640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc03806bdf0 0x7fc03806e2b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc050001eb0 tx=0x7fc050009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:35.202 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.198+0000 7fc04a7fc640 10 client.4295.objecter ms_handle_connect 0x7fc03806bdf0 2023-11-02T08:36:35.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.207+0000 7f03deffd640 1 -- 172.21.15.151:0/3416824893 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f03e00073d0 con 0x7f03f00b0ed0 2023-11-02T08:36:35.210 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.207+0000 7f03deffd640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f03d006bdc0 0x7f03d006e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:35.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.207+0000 7f03edd9b640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f03d006bdc0 0x7f03d006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:35.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.207+0000 7f03edd9b640 10 monclient: get_auth_request con 0x7f03d006bdc0 auth_method 0 2023-11-02T08:36:35.211 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.208+0000 7f03edd9b640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f03d006bdc0 0x7f03d006e280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03f013a360 tx=0x7f03e4006c80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:35.212 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.208+0000 7f03deffd640 10 client.4338.objecter ms_handle_connect 0x7f03d006bdc0 2023-11-02T08:36:35.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.209+0000 7fc04a7fc640 1 -- 172.21.15.151:0/3183502727 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc04407a280 con 0x7fc0541da620 2023-11-02T08:36:35.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.209+0000 7fc04a7fc640 10 client.4295.objecter ms_dispatch 0x7fc054000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:35.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.209+0000 7fc04a7fc640 3 client.4295.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:35.213 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.209+0000 7fc04a7fc640 3 client.4295.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:35.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.210+0000 7fc04a7fc640 20 client.4295.objecter dump_active .. 0 homeless 2023-11-02T08:36:35.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.210+0000 7fc04a7fc640 1 -- 172.21.15.151:0/3183502727 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fc044075ec0 con 0x7fc0541da620 2023-11-02T08:36:35.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.210+0000 7fc04a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.214 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.210+0000 7fc04a7fc640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.220+0000 7f03deffd640 1 -- 172.21.15.151:0/3416824893 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f03e0057630 con 0x7f03f00b0ed0 2023-11-02T08:36:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.220+0000 7f03deffd640 10 client.4338.objecter ms_dispatch 0x7f03f0000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:35.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.220+0000 7f03deffd640 3 client.4338.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:35.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.220+0000 7f03deffd640 3 client.4338.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:35.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.220+0000 7f03deffd640 20 client.4338.objecter dump_active .. 0 homeless 2023-11-02T08:36:35.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.220+0000 7f03deffd640 1 -- 172.21.15.151:0/3416824893 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f03e0053970 con 0x7f03f00b0ed0 2023-11-02T08:36:35.224 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.220+0000 7f03deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.225 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.220+0000 7f03deffd640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:35.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 1 -- 172.21.15.151:0/2921646097 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7000cb40 con 0x7f3f780b4570 2023-11-02T08:36:35.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:35.231 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 10 monclient: dump: 2023-11-02T08:36:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.233 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:35.233 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.233 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:35.233 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:35.234 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:35.234 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:35.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:35.234 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:35.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 1 -- 172.21.15.151:0/2921646097 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7000ccc0 con 0x7f3f780b4570 2023-11-02T08:36:35.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:35.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 1 -- 172.21.15.151:0/2921646097 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70014680 con 0x7f3f780b4570 2023-11-02T08:36:35.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:35.235 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:35.236 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6e7fc640 10 monclient: dump: 2023-11-02T08:36:35.236 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:35.236 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:35.236 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.236 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.237 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:35.237 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.237 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:35.237 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:35.237 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:35.238 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.238 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f6ffff640 4 set_mon_vals no callback set 2023-11-02T08:36:35.238 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f7ff49640 5 monclient: authenticate success, global_id 4298 2023-11-02T08:36:35.238 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f7ff49640 10 monclient: _renew_subs 2023-11-02T08:36:35.238 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f7ff49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:35.239 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f780ab3b0 con 0x7f3f780b4570 2023-11-02T08:36:35.239 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f7ff49640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:35.239 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f7ff49640 10 monclient: _renew_subs 2023-11-02T08:36:35.239 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f7ff49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:35.239 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f78142400 con 0x7f3f780b4570 2023-11-02T08:36:35.240 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.227+0000 7f3f7ff49640 1 librados: init done 2023-11-02T08:36:35.240 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.229+0000 7f3f7ff49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.240 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.229+0000 7f3f7ff49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.240 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.229+0000 7f3f7ff49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:35.241 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.229+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f781da230 con 0x7f3f780b4570 2023-11-02T08:36:35.241 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.237+0000 7f9714bab640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-02T08:36:35.242 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.237+0000 7f9714bab640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-02T08:36:35.242 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.237+0000 7f9714bab640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:35.242 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.237+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9710067920 con 0x7f97100b0ed0 2023-11-02T08:36:35.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.259+0000 7f3f6e7fc640 1 -- 172.21.15.151:0/2921646097 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3f7001d3b0 con 0x7f3f780b4570 2023-11-02T08:36:35.263 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.260+0000 7f3f6e7fc640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3f5c06bdf0 0x7f3f5c06e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.260+0000 7f3f7dce5640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3f5c06bdf0 0x7f3f5c06e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.260+0000 7f3f7dce5640 10 monclient: get_auth_request con 0x7f3f5c06bdf0 auth_method 0 2023-11-02T08:36:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.260+0000 7f3f6e7fc640 10 client.4298.objecter ms_handle_connect 0x7f3f5c06bdf0 2023-11-02T08:36:35.264 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.260+0000 7f3f7dce5640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3f5c06bdf0 0x7f3f5c06e2b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3f74035000 tx=0x7f3f74030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.319+0000 7faf35ffb640 1 -- 172.21.15.151:0/3515246389 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faf3c04e090 con 0x7faf40069ee0 2023-11-02T08:36:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.320+0000 7faf35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-11-02T08:36:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.320+0000 7faf35ffb640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:35.324 INFO:teuthology.orchestra.run.smithi151.stdout:60129542150 2023-11-02T08:36:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 10 librados: watch_flush enter 2023-11-02T08:36:35.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 10 librados: watch_flush exit 2023-11-02T08:36:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 20 client.4323.objecter shutdown clearing up homeless session... 2023-11-02T08:36:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 10 client.4323.objecter successfully canceled tick 2023-11-02T08:36:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7faf2406bdc0 msgr2=0x7faf2406e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7faf2406bdc0 0x7faf2406e280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faf4013e3e0 tx=0x7faf30006c90 comp rx=0 tx=0).stop 2023-11-02T08:36:35.326 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 10 monclient: shutdown 2023-11-02T08:36:35.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:35.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 msgr2=0x7faf401ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 0x7faf401ca140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf3c000c00 tx=0x7faf3c0046c0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 shutdown_connections 2023-11-02T08:36:35.327 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faf401419d0 0x7faf4013c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7faf2406bdc0 0x7faf2406e280 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4013e7f0 0x7faf4013ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 --2- 172.21.15.151:0/3515246389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faf40069ee0 0x7faf401ca140 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.322+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 >> 172.21.15.151:0/3515246389 conn(0x7faf400b3fa0 msgr2=0x7faf400b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.323+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 shutdown_connections 2023-11-02T08:36:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.323+0000 7faf476ca640 1 -- 172.21.15.151:0/3515246389 wait complete. 2023-11-02T08:36:35.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.323+0000 7faf476ca640 1 librados: shutdown 2023-11-02T08:36:35.335 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.332+0000 7fc5f97fa640 1 -- 172.21.15.151:0/3474826060 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc5f403d7a0 con 0x7fc5fc1d6260 2023-11-02T08:36:35.336 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.332+0000 7fc5f97fa640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc5e006bdc0 0x7fc5e006e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:35.336 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.332+0000 7fc600a10640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc5e006bdc0 0x7fc5e006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:35.336 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.332+0000 7fc600a10640 10 monclient: get_auth_request con 0x7fc5e006bdc0 auth_method 0 2023-11-02T08:36:35.337 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.333+0000 7fc5f97fa640 10 client.4381.objecter ms_handle_connect 0x7fc5e006bdc0 2023-11-02T08:36:35.337 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.333+0000 7fc600a10640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc5e006bdc0 0x7fc5e006e280 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00014e0 tx=0x7fc5f0007990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:35.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.2 2023-11-02T08:36:35.338 DEBUG:teuthology.parallel:result is None 2023-11-02T08:36:35.339 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.335+0000 7fc5f97fa640 1 -- 172.21.15.151:0/3474826060 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc5f407ab70 con 0x7fc5fc1d6260 2023-11-02T08:36:35.339 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.335+0000 7fc5f97fa640 10 client.4381.objecter ms_dispatch 0x7fc5fc000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:35.339 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.335+0000 7fc5f97fa640 3 client.4381.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:35.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.335+0000 7fc5f97fa640 3 client.4381.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:35.340 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.336+0000 7fc5f97fa640 20 client.4381.objecter dump_active .. 0 homeless 2023-11-02T08:36:35.348 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.345+0000 7fc5f97fa640 1 -- 172.21.15.151:0/3474826060 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7fc5f40767b0 con 0x7fc5fc1d6260 2023-11-02T08:36:35.349 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.345+0000 7fc5f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.349 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.345+0000 7fc5f97fa640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:35.428 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.425+0000 7fc05be84640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-02T08:36:35.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.425+0000 7fc05be84640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-02T08:36:35.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.425+0000 7fc05be84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:35.429 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.425+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc054067920 con 0x7fc0541da620 2023-11-02T08:36:35.433 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.430+0000 7f03f4f18640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-02T08:36:35.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.430+0000 7f03f4f18640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-02T08:36:35.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.430+0000 7f03f4f18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:36:35.434 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.430+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f03f00b6a30 con 0x7f03f00b0ed0 2023-11-02T08:36:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.434+0000 7f03deffd640 1 -- 172.21.15.151:0/3416824893 <== mon.0 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f03e0017020 con 0x7f03f00b0ed0 2023-11-02T08:36:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.434+0000 7f03deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-11-02T08:36:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.434+0000 7f03deffd640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:35.437 INFO:teuthology.orchestra.run.smithi151.stdout:60129542150 2023-11-02T08:36:35.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 10 librados: watch_flush enter 2023-11-02T08:36:35.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 10 librados: watch_flush exit 2023-11-02T08:36:35.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 20 client.4338.objecter shutdown clearing up homeless session... 2023-11-02T08:36:35.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 10 client.4338.objecter successfully canceled tick 2023-11-02T08:36:35.440 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f03d006bdc0 msgr2=0x7f03d006e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f03d006bdc0 0x7f03d006e280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03f013a360 tx=0x7f03e4006c80 comp rx=0 tx=0).stop 2023-11-02T08:36:35.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 10 monclient: shutdown 2023-11-02T08:36:35.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:35.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f00b0ed0 msgr2=0x7f03f0141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.441 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f00b0ed0 0x7f03f0141510 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03e000c4a0 tx=0x7f03e000c960 comp rx=0 tx=0).stop 2023-11-02T08:36:35.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 shutdown_connections 2023-11-02T08:36:35.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f03d006bdc0 0x7f03d006e280 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03f013e520 0x7f03f00acb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f03f00b4570 0x7f03f00ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 --2- 172.21.15.151:0/3416824893 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f03f00b0ed0 0x7f03f0141510 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.443 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 >> 172.21.15.151:0/3416824893 conn(0x7f03f0065d50 msgr2=0x7f03f00b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:35.443 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 shutdown_connections 2023-11-02T08:36:35.443 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 -- 172.21.15.151:0/3416824893 wait complete. 2023-11-02T08:36:35.443 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.436+0000 7f03f4f18640 1 librados: shutdown 2023-11-02T08:36:35.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.443+0000 7f96feffd640 1 -- 172.21.15.151:0/4262689732 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f970004e020 con 0x7f97100b0ed0 2023-11-02T08:36:35.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.443+0000 7f96feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-11-02T08:36:35.446 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.443+0000 7f96feffd640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:35.447 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 10 librados: watch_flush enter 2023-11-02T08:36:35.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 10 librados: watch_flush exit 2023-11-02T08:36:35.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 20 client.4378.objecter shutdown clearing up homeless session... 2023-11-02T08:36:35.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 10 client.4378.objecter successfully canceled tick 2023-11-02T08:36:35.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f96f006be40 msgr2=0x7f96f006e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f96f006be40 0x7f96f006e300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f97101d4220 tx=0x7f9704006c80 comp rx=0 tx=0).stop 2023-11-02T08:36:35.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 10 monclient: shutdown 2023-11-02T08:36:35.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:35.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b0ed0 msgr2=0x7f97100ab4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b0ed0 0x7f97100ab4c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f970002b820 tx=0x7f97000046c0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 shutdown_connections 2023-11-02T08:36:35.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97100b4570 0x7f9710138650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97100b1cb0 0x7f97100abf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f96f006be40 0x7f96f006e300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 --2- 172.21.15.151:0/4262689732 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97100b0ed0 0x7f97100ab4c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 >> 172.21.15.151:0/4262689732 conn(0x7f9710065d50 msgr2=0x7f971013ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:35.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 shutdown_connections 2023-11-02T08:36:35.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.445+0000 7f9714bab640 1 -- 172.21.15.151:0/4262689732 wait complete. 2023-11-02T08:36:35.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.446+0000 7f9714bab640 1 librados: shutdown 2023-11-02T08:36:35.454 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2023-11-02T08:36:35.454 DEBUG:teuthology.parallel:result is None 2023-11-02T08:36:35.464 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.4 2023-11-02T08:36:35.464 DEBUG:teuthology.parallel:result is None 2023-11-02T08:36:35.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.543+0000 7f3f6e7fc640 1 -- 172.21.15.151:0/2921646097 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3f7000ce30 con 0x7f3f780b4570 2023-11-02T08:36:35.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.543+0000 7f3f6e7fc640 10 client.4298.objecter ms_dispatch 0x7f3f78000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:35.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.543+0000 7f3f6e7fc640 3 client.4298.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:35.547 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.543+0000 7f3f6e7fc640 3 client.4298.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:35.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.543+0000 7f3f6e7fc640 20 client.4298.objecter dump_active .. 0 homeless 2023-11-02T08:36:35.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.543+0000 7f3f6e7fc640 1 -- 172.21.15.151:0/2921646097 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f3f700542e0 con 0x7f3f780b4570 2023-11-02T08:36:35.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.543+0000 7f3f6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.548 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.543+0000 7f3f6e7fc640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:35.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.550+0000 7fdea57fa640 1 -- 172.21.15.151:0/2086077792 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdea0043090 con 0x7fdea813e520 2023-11-02T08:36:35.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.550+0000 7fdea57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-11-02T08:36:35.553 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.550+0000 7fdea57fa640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:35.554 INFO:teuthology.orchestra.run.smithi151.stdout:55834574852 2023-11-02T08:36:35.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 10 librados: watch_flush enter 2023-11-02T08:36:35.555 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 10 librados: watch_flush exit 2023-11-02T08:36:35.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 20 client.4289.objecter shutdown clearing up homeless session... 2023-11-02T08:36:35.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 10 client.4289.objecter successfully canceled tick 2023-11-02T08:36:35.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fde8c06bdc0 msgr2=0x7fde8c06e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.556 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fde8c06bdc0 0x7fde8c06e280 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde9c001560 tx=0x7fde9c007990 comp rx=0 tx=0).stop 2023-11-02T08:36:35.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 10 monclient: shutdown 2023-11-02T08:36:35.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:35.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 msgr2=0x7fdea81d9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea81d9bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009fa0 tx=0x7fdea0004a40 comp rx=0 tx=0).stop 2023-11-02T08:36:35.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 shutdown_connections 2023-11-02T08:36:35.557 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdea813e520 0x7fdea81d9bf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fde8c06bdc0 0x7fde8c06e280 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdea80b4570 0x7fdea8141910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 --2- 172.21.15.151:0/2086077792 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdea80b0ed0 0x7fdea8141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 >> 172.21.15.151:0/2086077792 conn(0x7fdea8065d50 msgr2=0x7fdea813dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:35.558 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 shutdown_connections 2023-11-02T08:36:35.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 -- 172.21.15.151:0/2086077792 wait complete. 2023-11-02T08:36:35.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.552+0000 7fdeaee0e640 1 librados: shutdown 2023-11-02T08:36:35.559 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.554+0000 7fc04a7fc640 1 -- 172.21.15.151:0/3183502727 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc044042090 con 0x7fc0541da620 2023-11-02T08:36:35.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.555+0000 7fc04a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-11-02T08:36:35.560 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.555+0000 7fc04a7fc640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:35.560 INFO:teuthology.orchestra.run.smithi151.stdout:60129542148 2023-11-02T08:36:35.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 10 librados: watch_flush enter 2023-11-02T08:36:35.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 10 librados: watch_flush exit 2023-11-02T08:36:35.562 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 20 client.4295.objecter shutdown clearing up homeless session... 2023-11-02T08:36:35.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 10 client.4295.objecter successfully canceled tick 2023-11-02T08:36:35.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc03806bdf0 msgr2=0x7fc03806e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc03806bdf0 0x7fc03806e2b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc050001eb0 tx=0x7fc050009040 comp rx=0 tx=0).stop 2023-11-02T08:36:35.563 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 10 monclient: shutdown 2023-11-02T08:36:35.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:35.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0541da620 msgr2=0x7fc054136a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.564 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.557+0000 7fc05be84640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0541da620 0x7fc054136a70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc04402bbc0 tx=0x7fc044001000 comp rx=0 tx=0).stop 2023-11-02T08:36:35.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.558+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 shutdown_connections 2023-11-02T08:36:35.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.558+0000 7fc05be84640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc03806bdf0 0x7fc03806e2b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.558+0000 7fc05be84640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc054143590 0x7fc054143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.565 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.558+0000 7fc05be84640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc0541da620 0x7fc054136a70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.558+0000 7fc05be84640 1 --2- 172.21.15.151:0/3183502727 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc0541d9c00 0x7fc054136530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.558+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 >> 172.21.15.151:0/3183502727 conn(0x7fc054065d50 msgr2=0x7fc054073580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:35.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.558+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 shutdown_connections 2023-11-02T08:36:35.566 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.558+0000 7fc05be84640 1 -- 172.21.15.151:0/3183502727 wait complete. 2023-11-02T08:36:35.567 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.558+0000 7fc05be84640 1 librados: shutdown 2023-11-02T08:36:35.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2023-11-02T08:36:35.570 DEBUG:teuthology.parallel:result is None 2023-11-02T08:36:35.573 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.3 2023-11-02T08:36:35.574 DEBUG:teuthology.parallel:result is None 2023-11-02T08:36:35.589 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.586+0000 7fc602c74640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-02T08:36:35.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.586+0000 7fc602c74640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-02T08:36:35.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.586+0000 7fc602c74640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:35.590 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.586+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc5fc142620 con 0x7fc5fc1d6260 2023-11-02T08:36:35.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.590+0000 7fc5f97fa640 1 -- 172.21.15.151:0/3474826060 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc5fc142620 con 0x7fc5fc1d6260 2023-11-02T08:36:35.593 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.590+0000 7fc5f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-11-02T08:36:35.594 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.590+0000 7fc5f97fa640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:35.594 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:35.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 10 librados: watch_flush enter 2023-11-02T08:36:35.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 10 librados: watch_flush exit 2023-11-02T08:36:35.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 20 client.4381.objecter shutdown clearing up homeless session... 2023-11-02T08:36:35.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 10 client.4381.objecter successfully canceled tick 2023-11-02T08:36:35.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc5e006bdc0 msgr2=0x7fc5e006e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc5e006bdc0 0x7fc5e006e280 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00014e0 tx=0x7fc5f0007990 comp rx=0 tx=0).stop 2023-11-02T08:36:35.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 10 monclient: shutdown 2023-11-02T08:36:35.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:35.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5fc1d6260 msgr2=0x7fc5fc1d66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5fc1d6260 0x7fc5fc1d66e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc5f402c800 tx=0x7fc5f4004b50 comp rx=0 tx=0).stop 2023-11-02T08:36:35.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 shutdown_connections 2023-11-02T08:36:35.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fc5e006bdc0 0x7fc5e006e280 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5fc1d6260 0x7fc5fc1d66e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5fc0ab560 0x7fc5fc0ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 --2- 172.21.15.151:0/3474826060 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5ec005380 0x7fc5fc0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 >> 172.21.15.151:0/3474826060 conn(0x7fc5fc065d50 msgr2=0x7fc5fc13c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:35.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 shutdown_connections 2023-11-02T08:36:35.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 -- 172.21.15.151:0/3474826060 wait complete. 2023-11-02T08:36:35.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.593+0000 7fc602c74640 1 librados: shutdown 2023-11-02T08:36:35.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.5 2023-11-02T08:36:35.611 DEBUG:teuthology.parallel:result is None 2023-11-02T08:36:35.744 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.741+0000 7f3f7ff49640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-02T08:36:35.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.741+0000 7f3f7ff49640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-02T08:36:35.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.741+0000 7f3f7ff49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.151:3301/0 2023-11-02T08:36:35.745 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.741+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3f78067920 con 0x7f3f780b4570 2023-11-02T08:36:35.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.745+0000 7f3f6e7fc640 1 -- 172.21.15.151:0/2921646097 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f70019070 con 0x7f3f780b4570 2023-11-02T08:36:35.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.745+0000 7f3f6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-11-02T08:36:35.748 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.745+0000 7f3f6e7fc640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:35.749 INFO:teuthology.orchestra.run.smithi151.stdout:60129542149 2023-11-02T08:36:35.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 10 librados: watch_flush enter 2023-11-02T08:36:35.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 10 librados: watch_flush exit 2023-11-02T08:36:35.751 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 20 client.4298.objecter shutdown clearing up homeless session... 2023-11-02T08:36:35.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 10 client.4298.objecter successfully canceled tick 2023-11-02T08:36:35.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3f5c06bdf0 msgr2=0x7f3f5c06e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.752 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3f5c06bdf0 0x7f3f5c06e2b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3f74035000 tx=0x7f3f74030000 comp rx=0 tx=0).stop 2023-11-02T08:36:35.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 10 monclient: shutdown 2023-11-02T08:36:35.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:35.753 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b4570 msgr2=0x7f3f781daa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b4570 0x7f3f781daa70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000b970 tx=0x7f3f7000be30 comp rx=0 tx=0).stop 2023-11-02T08:36:35.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 shutdown_connections 2023-11-02T08:36:35.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f780b4570 0x7f3f781daa70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f3f5c06bdf0 0x7f3f5c06e2b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.754 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f780b1cb0 0x7f3f781da530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 --2- 172.21.15.151:0/2921646097 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f780b0ed0 0x7f3f780ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 >> 172.21.15.151:0/2921646097 conn(0x7f3f78065d50 msgr2=0x7f3f7813ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:35.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 shutdown_connections 2023-11-02T08:36:35.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 -- 172.21.15.151:0/2921646097 wait complete. 2023-11-02T08:36:35.755 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.747+0000 7f3f7ff49640 1 librados: shutdown 2023-11-02T08:36:35.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.1 2023-11-02T08:36:35.762 DEBUG:teuthology.parallel:result is None 2023-11-02T08:36:35.763 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-02T08:36:35.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-02T08:36:35.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.860+0000 7f726824c640 10 monclient: get_monmap_and_config 2023-11-02T08:36:35.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.860+0000 7f726824c640 10 monclient: build_initial_monmap 2023-11-02T08:36:35.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.860+0000 7f726824c640 10 monclient: monmap: 2023-11-02T08:36:35.864 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:35.864 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:35.865 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:35.861823+0000 2023-11-02T08:36:35.865 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:35.861823+0000 2023-11-02T08:36:35.865 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:35.865 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.865 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:35.866 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:35.866 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:35.866 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:35.866 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:35.866 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.861+0000 7f726824c640 1 Processor -- start 2023-11-02T08:36:35.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.861+0000 7f726824c640 1 -- start start 2023-11-02T08:36:35.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.861+0000 7f726824c640 10 monclient: init 2023-11-02T08:36:35.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:35.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-11-02T08:36:35.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72600b4170 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:35.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f72600a8df0 0x7f7260141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:35.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 10 monclient(hunting): picked mon.noname-a con 0x7f72600a8df0 addr v2:172.21.15.151:3300/0 2023-11-02T08:36:35.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72600b13e0 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:35.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72600aa3d0 con 0x7f72600b4170 2023-11-02T08:36:35.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72600a92d0 con 0x7f72600b13e0 2023-11-02T08:36:35.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:35.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72600b4550 con 0x7f72600a8df0 2023-11-02T08:36:35.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:35.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f726824c640 10 monclient(hunting): authenticate will time out at 1367.218206s 2023-11-02T08:36:35.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f72657e7640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f72600a8df0 0x7f7260141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:35.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f72667e9640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f72600b13e0 0x7f726013e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:42940/0 (socket says 172.21.15.151:42940) 2023-11-02T08:36:35.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.862+0000 7f72667e9640 1 -- 172.21.15.151:0/376424047 learned_addr learned my addr 172.21.15.151:0/376424047 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:35.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.863+0000 7f72657e7640 10 monclient(hunting): get_auth_request con 0x7f72600a8df0 auth_method 0 2023-11-02T08:36:35.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.863+0000 7f72657e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:35.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.863+0000 7f72657e7640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:35.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.863+0000 7f72657e7640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:35.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.864+0000 7f72657e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:35.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.864+0000 7f72657e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:35.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.864+0000 7f72657e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:35.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.864+0000 7f7264fe6640 1 -- 172.21.15.151:0/376424047 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355811555 0 0) 0x7f72600a92d0 con 0x7f72600b13e0 2023-11-02T08:36:35.874 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.864+0000 7f7264fe6640 10 monclient(hunting): _negotiate 2023-11-02T08:36:35.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.864+0000 7f7264fe6640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:35.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.864+0000 7f7264fe6640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:35.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.864+0000 7f7264fe6640 10 monclient(hunting): my global_id is 4384 2023-11-02T08:36:35.875 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.864+0000 7f7264fe6640 1 -- 172.21.15.151:0/376424047 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f724c003520 con 0x7f72600b13e0 2023-11-02T08:36:35.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.866+0000 7f72657e7640 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2023-11-02T08:36:35.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.866+0000 7f72657e7640 1 -- 172.21.15.151:0/376424047 >> v1:172.21.15.178:6789/0 conn(0x7f72600b13e0 legacy=0x7f726013e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.876 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.866+0000 7f72657e7640 1 -- 172.21.15.151:0/376424047 >> v1:172.21.15.151:6789/0 conn(0x7f72600b4170 legacy=0x7f7260141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.866+0000 7f72657e7640 10 monclient: _finish_hunting 0 2023-11-02T08:36:35.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.866+0000 7f72657e7640 1 monclient: found mon.noname-a 2023-11-02T08:36:35.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.866+0000 7f72657e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:35.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.866+0000 7f72657e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:36:35.877 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.866+0000 7f72657e7640 1 -- 172.21.15.151:0/376424047 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7260142450 con 0x7f72600a8df0 2023-11-02T08:36:35.878 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.870+0000 7f72657e7640 1 --2- 172.21.15.151:0/376424047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 0x7f7260141f10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72500097d0 tx=0x7f725002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7989af08965fb6de server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:35.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 1 -- 172.21.15.151:0/376424047 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250031830 con 0x7f72600a8df0 2023-11-02T08:36:35.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:35.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:36:35.879 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 10 monclient: dump: 2023-11-02T08:36:35.880 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:35.880 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:35.880 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.880 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.880 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:35.881 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.881 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:35.881 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:35.881 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:35.881 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 10 monclient: _finish_auth 0 2023-11-02T08:36:35.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 10 monclient: _check_auth_tickets 2023-11-02T08:36:35.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:35.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 1 -- 172.21.15.151:0/376424047 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7250031990 con 0x7f72600a8df0 2023-11-02T08:36:35.882 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:35.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 1 -- 172.21.15.151:0/376424047 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250031d20 con 0x7f72600a8df0 2023-11-02T08:36:35.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 5 monclient: authenticate success, global_id 4341 2023-11-02T08:36:35.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:35.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 4 set_mon_vals no callback set 2023-11-02T08:36:35.883 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:35.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:36:35.884 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f7264fe6640 10 monclient: dump: 2023-11-02T08:36:35.884 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:35.884 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:35.884 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.885 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.885 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:35.885 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.885 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:35.885 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:35.885 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:35.886 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 10 monclient: shutdown 2023-11-02T08:36:35.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:35.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 1 -- 172.21.15.151:0/376424047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 msgr2=0x7f7260141f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 1 --2- 172.21.15.151:0/376424047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 0x7f7260141f10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72500097d0 tx=0x7f725002b8d0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 1 -- 172.21.15.151:0/376424047 shutdown_connections 2023-11-02T08:36:35.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 1 --2- 172.21.15.151:0/376424047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 0x7f7260141f10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 1 -- 172.21.15.151:0/376424047 >> 172.21.15.151:0/376424047 conn(0x7f7260065d80 msgr2=0x7f7260066190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:35.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 1 -- 172.21.15.151:0/376424047 shutdown_connections 2023-11-02T08:36:35.887 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.875+0000 7f726824c640 1 -- 172.21.15.151:0/376424047 wait complete. 2023-11-02T08:36:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.876+0000 7f726824c640 10 monclient: build_initial_monmap 2023-11-02T08:36:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.876+0000 7f726824c640 10 monclient: monmap: 2023-11-02T08:36:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:35.893 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:35.893 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:35.893 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.893 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:35.893 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:35.894 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:35.894 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.876+0000 7f726824c640 1 librados: starting msgr at 2023-11-02T08:36:35.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.876+0000 7f726824c640 1 librados: starting objecter 2023-11-02T08:36:35.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.876+0000 7f726824c640 1 Processor -- start 2023-11-02T08:36:35.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 1 -- start start 2023-11-02T08:36:35.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 1 librados: setting wanted keys 2023-11-02T08:36:35.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 1 librados: calling monclient init 2023-11-02T08:36:35.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient: init 2023-11-02T08:36:35.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:35.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient: _add_conns ranks=[0,1,2] 2023-11-02T08:36:35.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 0x7f72600adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:35.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient(hunting): picked mon.noname-a con 0x7f72600a8df0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:35.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72600b13e0 0x7f72601d6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:35.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient(hunting): picked mon.noname-b con 0x7f72600b13e0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:35.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72600b4170 0x7f72601d6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:35.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient(hunting): picked mon.noname-c con 0x7f72600b4170 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:35.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:35.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72600af4a0 con 0x7f72600a8df0 2023-11-02T08:36:35.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:35.897 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72600ad5d0 con 0x7f72600b13e0 2023-11-02T08:36:35.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:35.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726013ae90 con 0x7f72600b4170 2023-11-02T08:36:35.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:35.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.877+0000 7f726824c640 10 monclient(hunting): authenticate will time out at 1367.233452s 2023-11-02T08:36:35.898 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f7265fe8640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 0x7f72600adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:35.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72667e9640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72600b4170 0x7f72601d6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:35.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72657e7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72600b13e0 0x7f72601d6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:35.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f7265fe8640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 0x7f72600adb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:38848/0 (socket says 172.21.15.151:38848) 2023-11-02T08:36:35.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f7265fe8640 1 -- 172.21.15.151:0/1481172007 learned_addr learned my addr 172.21.15.151:0/1481172007 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:35.899 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f7265fe8640 10 monclient(hunting): get_auth_request con 0x7f72600a8df0 auth_method 0 2023-11-02T08:36:35.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f7265fe8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:35.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f7265fe8640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:35.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f7265fe8640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:35.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72657e7640 10 monclient(hunting): get_auth_request con 0x7f72600b13e0 auth_method 0 2023-11-02T08:36:35.900 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72657e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:35.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72657e7640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:35.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72657e7640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:35.901 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72667e9640 10 monclient(hunting): get_auth_request con 0x7f72600b4170 auth_method 0 2023-11-02T08:36:35.902 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72667e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:35.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72667e9640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:35.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.878+0000 7f72667e9640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:35.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.879+0000 7f72657e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:35.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.879+0000 7f72657e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:35.903 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.879+0000 7f72657e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:35.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.879+0000 7f7265fe8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:35.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.879+0000 7f7265fe8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:35.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.879+0000 7f7265fe8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:35.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.879+0000 7f72667e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:35.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.879+0000 7f72667e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:35.904 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.879+0000 7f72667e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:35.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2023-11-02T08:36:35.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 1 -- 172.21.15.151:0/1481172007 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72600b4170 msgr2=0x7f72601d6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72600b4170 0x7f72601d6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 1 -- 172.21.15.151:0/1481172007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 msgr2=0x7f72600adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:35.905 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 0x7f72600adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:35.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 10 monclient: _finish_hunting 0 2023-11-02T08:36:35.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 1 monclient: found mon.noname-b 2023-11-02T08:36:35.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:35.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:35.906 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.881+0000 7f72657e7640 1 -- 172.21.15.151:0/1481172007 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7250009400 con 0x7f72600b13e0 2023-11-02T08:36:35.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.883+0000 7f72657e7640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72600b13e0 0x7f72601d6830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7250033000 tx=0x7f72500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:35.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.883+0000 7f72567fc640 10 client.?.objecter ms_handle_connect 0x7f72600b13e0 2023-11-02T08:36:35.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.883+0000 7f72567fc640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:35.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 1 -- 172.21.15.151:0/1481172007 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725003b9f0 con 0x7f72600b13e0 2023-11-02T08:36:35.907 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:35.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:35.908 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 10 monclient: dump: 2023-11-02T08:36:35.908 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:35.908 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:35.908 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.909 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.909 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:35.909 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.909 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:35.909 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:35.909 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:35.910 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 10 monclient: _finish_auth 0 2023-11-02T08:36:35.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 10 monclient: _check_auth_tickets 2023-11-02T08:36:35.910 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:35.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 1 -- 172.21.15.151:0/1481172007 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725003bb90 con 0x7f72600b13e0 2023-11-02T08:36:35.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f726824c640 5 monclient: authenticate success, global_id 4387 2023-11-02T08:36:35.913 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f726824c640 10 monclient: _renew_subs 2023-11-02T08:36:35.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f726824c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:35.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72600ac490 con 0x7f72600b13e0 2023-11-02T08:36:35.914 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:35.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 1 -- 172.21.15.151:0/1481172007 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72500453f0 con 0x7f72600b13e0 2023-11-02T08:36:35.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:35.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:35.915 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f72567fc640 10 monclient: dump: 2023-11-02T08:36:35.915 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:35.916 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:35.916 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.916 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:35.916 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:35.916 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:35.916 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:35.917 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:35.917 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:35.917 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:35.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f7257fff640 4 set_mon_vals no callback set 2023-11-02T08:36:35.917 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f726824c640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:35.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f726824c640 10 monclient: _renew_subs 2023-11-02T08:36:35.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f726824c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:35.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72601d7490 con 0x7f72600b13e0 2023-11-02T08:36:35.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.887+0000 7f726824c640 1 librados: init done 2023-11-02T08:36:35.918 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.888+0000 7f726824c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.888+0000 7f726824c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:35.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.888+0000 7f726824c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:35.919 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.888+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726004e040 con 0x7f72600b13e0 2023-11-02T08:36:35.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.906+0000 7f72567fc640 1 -- 172.21.15.151:0/1481172007 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f725004e450 con 0x7f72600b13e0 2023-11-02T08:36:35.920 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.906+0000 7f72567fc640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f724406be10 0x7f724406e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:35.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.906+0000 7f7265fe8640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f724406be10 0x7f724406e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:35.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.906+0000 7f7265fe8640 10 monclient: get_auth_request con 0x7f724406be10 auth_method 0 2023-11-02T08:36:35.921 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.907+0000 7f7265fe8640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f724406be10 0x7f724406e2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f725c0017a0 tx=0x7f725c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:35.922 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.907+0000 7f72567fc640 10 client.4387.objecter ms_handle_connect 0x7f724406be10 2023-11-02T08:36:35.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.908+0000 7f72567fc640 1 -- 172.21.15.151:0/1481172007 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7250079f70 con 0x7f72600b13e0 2023-11-02T08:36:35.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.908+0000 7f72567fc640 10 client.4387.objecter ms_dispatch 0x7f7260000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:35.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.908+0000 7f72567fc640 3 client.4387.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:35.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.908+0000 7f72567fc640 3 client.4387.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:35.923 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:35.908+0000 7f72567fc640 20 client.4387.objecter dump_active .. 0 homeless 2023-11-02T08:36:36.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.148+0000 7f72567fc640 1 -- 172.21.15.151:0/1481172007 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f72500762b0 con 0x7f72600b13e0 2023-11-02T08:36:36.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.148+0000 7f72567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:36.151 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.148+0000 7f72567fc640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:36.323 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.320+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 --> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72600a84b0 con 0x7f724406be10 2023-11-02T08:36:36.325 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.322+0000 7f72567fc640 1 -- 172.21.15.151:0/1481172007 <== mgr.4108 v2:172.21.15.178:6800/85498 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48034 (secure 0 0 0) 0x7f72600a84b0 con 0x7f724406be10 2023-11-02T08:36:36.326 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:36:36.327 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2023-11-02T08:36:36.328 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.324+0000 7f726824c640 10 librados: watch_flush enter 2023-11-02T08:36:36.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 10 librados: watch_flush exit 2023-11-02T08:36:36.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 20 client.4387.objecter shutdown clearing up homeless session... 2023-11-02T08:36:36.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 10 client.4387.objecter successfully canceled tick 2023-11-02T08:36:36.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f724406be10 msgr2=0x7f724406e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:36.329 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f724406be10 0x7f724406e2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f725c0017a0 tx=0x7f725c009040 comp rx=0 tx=0).stop 2023-11-02T08:36:36.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 10 monclient: shutdown 2023-11-02T08:36:36.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:36.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72600b13e0 msgr2=0x7f72601d6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:36.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72600b13e0 0x7f72601d6830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7250033000 tx=0x7f72500046c0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.330 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 shutdown_connections 2023-11-02T08:36:36.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72600b4170 0x7f72601d6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f724406be10 0x7f724406e2d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72600b13e0 0x7f72601d6830 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 --2- 172.21.15.151:0/1481172007 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72600a8df0 0x7f72600adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.331 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 >> 172.21.15.151:0/1481172007 conn(0x7f7260065d80 msgr2=0x7f7260066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:36.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 shutdown_connections 2023-11-02T08:36:36.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 -- 172.21.15.151:0/1481172007 wait complete. 2023-11-02T08:36:36.332 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.325+0000 7f726824c640 1 librados: shutdown 2023-11-02T08:36:36.339 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-11-02T08:36:34.941234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":101,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218160,"kb_used_data":2544,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754756560,"statfs":{"total":773094113280,"available":772870717440,"internally_reserved":0,"allocated":2605056,"data_stored":1409152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":459280,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":77,"num_write":94,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.118940"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.443027+0000","last_change":"2023-11-02T08:36:29.169375+0000","last_active":"2023-11-02T08:36:30.443027+0000","last_peered":"2023-11-02T08:36:30.443027+0000","last_clean":"2023-11-02T08:36:30.443027+0000","last_became_active":"2023-11-02T08:36:29.139473+0000","last_became_peered":"2023-11-02T08:36:29.139473+0000","last_unstale":"2023-11-02T08:36:30.443027+0000","last_undegraded":"2023-11-02T08:36:30.443027+0000","last_fullsized":"2023-11-02T08:36:30.443027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T18:14:09.805168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.660528+0000","last_change":"2023-11-02T08:36:30.654204+0000","last_active":"2023-11-02T08:36:30.660528+0000","last_peered":"2023-11-02T08:36:30.660528+0000","last_clean":"2023-11-02T08:36:30.660528+0000","last_became_active":"2023-11-02T08:36:30.648476+0000","last_became_peered":"2023-11-02T08:36:30.648476+0000","last_unstale":"2023-11-02T08:36:30.660528+0000","last_undegraded":"2023-11-02T08:36:30.660528+0000","last_fullsized":"2023-11-02T08:36:30.660528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T14:11:58.899468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.926369+0000","last_change":"2023-11-02T08:36:28.981877+0000","last_active":"2023-11-02T08:36:30.926369+0000","last_peered":"2023-11-02T08:36:30.926369+0000","last_clean":"2023-11-02T08:36:30.926369+0000","last_became_active":"2023-11-02T08:36:28.976660+0000","last_became_peered":"2023-11-02T08:36:28.976660+0000","last_unstale":"2023-11-02T08:36:30.926369+0000","last_undegraded":"2023-11-02T08:36:30.926369+0000","last_fullsized":"2023-11-02T08:36:30.926369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T19:12:52.578679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.630882+0000","last_change":"2023-11-02T08:36:30.611750+0000","last_active":"2023-11-02T08:36:30.630882+0000","last_peered":"2023-11-02T08:36:30.630882+0000","last_clean":"2023-11-02T08:36:30.630882+0000","last_became_active":"2023-11-02T08:36:30.583185+0000","last_became_peered":"2023-11-02T08:36:30.583185+0000","last_unstale":"2023-11-02T08:36:30.630882+0000","last_undegraded":"2023-11-02T08:36:30.630882+0000","last_fullsized":"2023-11-02T08:36:30.630882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T14:18:01.847932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:31.105423+0000","last_change":"2023-11-02T08:36:29.148625+0000","last_active":"2023-11-02T08:36:31.105423+0000","last_peered":"2023-11-02T08:36:31.105423+0000","last_clean":"2023-11-02T08:36:31.105423+0000","last_became_active":"2023-11-02T08:36:29.121362+0000","last_became_peered":"2023-11-02T08:36:29.121362+0000","last_unstale":"2023-11-02T08:36:31.105423+0000","last_undegraded":"2023-11-02T08:36:31.105423+0000","last_fullsized":"2023-11-02T08:36:31.105423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T19:40:36.987496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.627752+0000","last_change":"2023-11-02T08:36:30.029990+0000","last_active":"2023-11-02T08:36:30.627752+0000","last_peered":"2023-11-02T08:36:30.627752+0000","last_clean":"2023-11-02T08:36:30.627752+0000","last_became_active":"2023-11-02T08:36:29.989223+0000","last_became_peered":"2023-11-02T08:36:29.989223+0000","last_unstale":"2023-11-02T08:36:30.627752+0000","last_undegraded":"2023-11-02T08:36:30.627752+0000","last_fullsized":"2023-11-02T08:36:30.627752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T16:23:35.917732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.918286+0000","last_change":"2023-11-02T08:36:29.095539+0000","last_active":"2023-11-02T08:36:30.918286+0000","last_peered":"2023-11-02T08:36:30.918286+0000","last_clean":"2023-11-02T08:36:30.918286+0000","last_became_active":"2023-11-02T08:36:29.090475+0000","last_became_peered":"2023-11-02T08:36:29.090475+0000","last_unstale":"2023-11-02T08:36:30.918286+0000","last_undegraded":"2023-11-02T08:36:30.918286+0000","last_fullsized":"2023-11-02T08:36:30.918286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T09:01:37.136232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:31.107475+0000","last_change":"2023-11-02T08:36:29.175364+0000","last_active":"2023-11-02T08:36:31.107475+0000","last_peered":"2023-11-02T08:36:31.107475+0000","last_clean":"2023-11-02T08:36:31.107475+0000","last_became_active":"2023-11-02T08:36:29.169756+0000","last_became_peered":"2023-11-02T08:36:29.169756+0000","last_unstale":"2023-11-02T08:36:31.107475+0000","last_undegraded":"2023-11-02T08:36:31.107475+0000","last_fullsized":"2023-11-02T08:36:31.107475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T13:25:39.845560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'65","reported_seq":106,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-02T08:36:34.729115+0000","last_change":"2023-11-02T08:36:24.330685+0000","last_active":"2023-11-02T08:36:34.729115+0000","last_peered":"2023-11-02T08:36:34.729115+0000","last_clean":"2023-11-02T08:36:34.729115+0000","last_became_active":"2023-11-02T08:36:24.298799+0000","last_became_peered":"2023-11-02T08:36:24.298799+0000","last_unstale":"2023-11-02T08:36:34.729115+0000","last_undegraded":"2023-11-02T08:36:34.729115+0000","last_fullsized":"2023-11-02T08:36:34.729115+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:21.736705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:21.736705+0000","last_clean_scrub_stamp":"2023-11-02T08:36:21.736705+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T16:28:14.976849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":101,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":101,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":626688,"data_stored":474524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.476}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4560000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3460000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.967000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.382999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.757999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4710000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.98}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.587}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.105}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.148}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.047000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5259999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.162000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.273}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.832000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.936999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.516999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5869999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4930000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.195}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.756}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3719999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.18}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.34}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8420000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.776}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.818000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.343}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.404999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.654}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":39.683999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.058999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.189}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.817}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.838999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.667999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.739999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1070000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7089999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.885999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.376000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":61.694000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.211}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.936}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":626688,"data_stored":474524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":65.715000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":151.25700000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0220000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.039999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":120.617}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":124.18899999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":32.896000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-02T08:36:36.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-02T08:36:36.442 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.439+0000 7f1ba44e7640 10 monclient: get_monmap_and_config 2023-11-02T08:36:36.443 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.439+0000 7f1ba44e7640 10 monclient: build_initial_monmap 2023-11-02T08:36:36.443 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.439+0000 7f1ba44e7640 10 monclient: monmap: 2023-11-02T08:36:36.443 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:36.443 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:36.444 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:36.440547+0000 2023-11-02T08:36:36.444 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:36.440547+0000 2023-11-02T08:36:36.444 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:36.444 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:36.444 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:36.445 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:36.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 1 Processor -- start 2023-11-02T08:36:36.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 1 -- start start 2023-11-02T08:36:36.448 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient: init 2023-11-02T08:36:36.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:36.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-11-02T08:36:36.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b9c0b4170 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:36.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1b9c0a8df0 0x7f1b9c141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:36.449 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient(hunting): picked mon.noname-c con 0x7f1b9c0a8df0 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:36.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0b13e0 0x7f1b9c13e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:36.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient(hunting): picked mon.noname-b con 0x7f1b9c0b13e0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:36.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b9c0aa3d0 con 0x7f1b9c0b4170 2023-11-02T08:36:36.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:36.450 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9c13f420 con 0x7f1b9c0a8df0 2023-11-02T08:36:36.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:36.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0b4550 con 0x7f1b9c0b13e0 2023-11-02T08:36:36.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:36.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.440+0000 7f1ba44e7640 10 monclient(hunting): authenticate will time out at 1367.796719s 2023-11-02T08:36:36.451 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.441+0000 7f1ba2a84640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0b13e0 0x7f1b9c13e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:36.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.441+0000 7f1ba1a82640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1b9c0a8df0 0x7f1b9c141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:36.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.441+0000 7f1ba2283640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f1b9c0b4170 0x7f1b9c141800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.151:42942/0 (socket says 172.21.15.151:42942) 2023-11-02T08:36:36.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.441+0000 7f1ba2283640 1 -- 172.21.15.151:0/203023407 learned_addr learned my addr 172.21.15.151:0/203023407 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:36.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.441+0000 7f1ba2a84640 10 monclient(hunting): get_auth_request con 0x7f1b9c0b13e0 auth_method 0 2023-11-02T08:36:36.452 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.441+0000 7f1ba2a84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:36.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.441+0000 7f1ba2a84640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:36.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.441+0000 7f1ba2a84640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:36.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba1a82640 10 monclient(hunting): get_auth_request con 0x7f1b9c0a8df0 auth_method 0 2023-11-02T08:36:36.453 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba1a82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:36.455 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba1a82640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:36.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba1a82640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:36.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba2a84640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:36.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba2a84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:36.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba2a84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:36.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba1a82640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:36.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba1a82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:36.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.442+0000 7f1ba1a82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:36.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.443+0000 7f1ba2a84640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2023-11-02T08:36:36.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.443+0000 7f1ba2a84640 1 -- 172.21.15.151:0/203023407 >> v2:172.21.15.178:3300/0 conn(0x7f1b9c0a8df0 msgr2=0x7f1b9c141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:36:36.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.443+0000 7f1ba2a84640 1 --2- 172.21.15.151:0/203023407 >> v2:172.21.15.178:3300/0 conn(0x7f1b9c0a8df0 0x7f1b9c141f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.443+0000 7f1ba2a84640 1 -- 172.21.15.151:0/203023407 >> v1:172.21.15.178:6789/0 conn(0x7f1b9c0b4170 legacy=0x7f1b9c141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:36.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.443+0000 7f1ba2a84640 10 monclient: _finish_hunting 0 2023-11-02T08:36:36.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.443+0000 7f1ba2a84640 1 monclient: found mon.noname-b 2023-11-02T08:36:36.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.443+0000 7f1ba2a84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:36.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.443+0000 7f1ba2a84640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.151:3301/0 2023-11-02T08:36:36.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.443+0000 7f1ba2a84640 1 -- 172.21.15.151:0/203023407 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c0a92d0 con 0x7f1b9c0b13e0 2023-11-02T08:36:36.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.445+0000 7f1ba2a84640 1 --2- 172.21.15.151:0/203023407 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0b13e0 0x7f1b9c13e340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1b980097d0 tx=0x7f1b9802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be949f03a457d3d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:36.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 1 -- 172.21.15.151:0/203023407 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98031770 con 0x7f1b9c0b13e0 2023-11-02T08:36:36.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:36.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:36.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 10 monclient: dump: 2023-11-02T08:36:36.464 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:36.464 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:36.464 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:36.464 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:36.464 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:36.465 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:36.465 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:36.465 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:36.465 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:36.465 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:36.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 10 monclient: _finish_auth 0 2023-11-02T08:36:36.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 10 monclient: _check_auth_tickets 2023-11-02T08:36:36.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:36.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 1 -- 172.21.15.151:0/203023407 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b980318d0 con 0x7f1b9c0b13e0 2023-11-02T08:36:36.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:36.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 1 -- 172.21.15.151:0/203023407 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98031c60 con 0x7f1b9c0b13e0 2023-11-02T08:36:36.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:36.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-11-02T08:36:36.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba1281640 10 monclient: dump: 2023-11-02T08:36:36.467 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:36.468 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:36.468 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:36.468 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:36.468 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:36.468 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:36.469 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:36.471 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:36.471 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:36.471 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:36.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba44e7640 5 monclient: authenticate success, global_id 4304 2023-11-02T08:36:36.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba44e7640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:36.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.449+0000 7f1ba44e7640 4 set_mon_vals no callback set 2023-11-02T08:36:36.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 10 monclient: shutdown 2023-11-02T08:36:36.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 1 -- 172.21.15.151:0/203023407 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0b13e0 msgr2=0x7f1b9c13e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 1 --2- 172.21.15.151:0/203023407 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0b13e0 0x7f1b9c13e340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1b980097d0 tx=0x7f1b9802b810 comp rx=0 tx=0).stop 2023-11-02T08:36:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 1 -- 172.21.15.151:0/203023407 shutdown_connections 2023-11-02T08:36:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 1 --2- 172.21.15.151:0/203023407 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0b13e0 0x7f1b9c13e340 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 1 --2- 172.21.15.151:0/203023407 >> v2:172.21.15.178:3300/0 conn(0x7f1b9c0a8df0 0x7f1b9c141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 1 -- 172.21.15.151:0/203023407 >> 172.21.15.151:0/203023407 conn(0x7f1b9c065d80 msgr2=0x7f1b9c066190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:36.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 1 -- 172.21.15.151:0/203023407 shutdown_connections 2023-11-02T08:36:36.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.450+0000 7f1ba44e7640 1 -- 172.21.15.151:0/203023407 wait complete. 2023-11-02T08:36:36.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.451+0000 7f1ba44e7640 10 monclient: build_initial_monmap 2023-11-02T08:36:36.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.451+0000 7f1ba44e7640 10 monclient: monmap: 2023-11-02T08:36:36.475 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:36.475 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:36.475 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:36.475 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:36.476 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:36.476 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:36.476 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:36.476 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:36.477 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:36.477 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:36.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.451+0000 7f1ba44e7640 1 librados: starting msgr at 2023-11-02T08:36:36.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.451+0000 7f1ba44e7640 1 librados: starting objecter 2023-11-02T08:36:36.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.451+0000 7f1ba44e7640 1 Processor -- start 2023-11-02T08:36:36.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.451+0000 7f1ba44e7640 1 -- start start 2023-11-02T08:36:36.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.451+0000 7f1ba44e7640 1 librados: setting wanted keys 2023-11-02T08:36:36.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.451+0000 7f1ba44e7640 1 librados: calling monclient init 2023-11-02T08:36:36.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.451+0000 7f1ba44e7640 10 monclient: init 2023-11-02T08:36:36.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:36.478 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-02T08:36:36.479 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0a8df0 0x7f1b9c145580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:36.479 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient(hunting): picked mon.noname-c con 0x7f1b9c0a8df0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:36.479 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b9c0b13e0 0x7f1b9c0ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:36.479 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient(hunting): picked mon.noname-a con 0x7f1b9c0b13e0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:36.479 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b9c0b4170 0x7f1b9c1d96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:36.480 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient(hunting): picked mon.noname-b con 0x7f1b9c0b4170 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:36.480 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:36.480 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0adcf0 con 0x7f1b9c0b13e0 2023-11-02T08:36:36.480 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:36.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0ae6f0 con 0x7f1b9c0b4170 2023-11-02T08:36:36.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:36.481 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0ac0d0 con 0x7f1b9c0a8df0 2023-11-02T08:36:36.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:36.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba44e7640 10 monclient(hunting): authenticate will time out at 1367.807984s 2023-11-02T08:36:36.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2283640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0a8df0 0x7f1b9c145580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:36.482 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2a84640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b9c0b4170 0x7f1b9c1d96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:36.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba1a82640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b9c0b13e0 0x7f1b9c0ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:36.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2a84640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b9c0b4170 0x7f1b9c1d96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36586/0 (socket says 172.21.15.151:36586) 2023-11-02T08:36:36.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2a84640 1 -- 172.21.15.151:0/2798847971 learned_addr learned my addr 172.21.15.151:0/2798847971 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:36.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2a84640 10 monclient(hunting): get_auth_request con 0x7f1b9c0b4170 auth_method 0 2023-11-02T08:36:36.483 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2a84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:36.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2a84640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:36.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2a84640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:36.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2283640 10 monclient(hunting): get_auth_request con 0x7f1b9c0a8df0 auth_method 0 2023-11-02T08:36:36.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2283640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:36.484 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2283640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:36.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.452+0000 7f1ba2283640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:36.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba1a82640 10 monclient(hunting): get_auth_request con 0x7f1b9c0b13e0 auth_method 0 2023-11-02T08:36:36.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba1a82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:36.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba1a82640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:36.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba1a82640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:36.485 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba2a84640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:36.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba2a84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:36.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba2a84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:36.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba2283640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:36.486 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba2283640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:36.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba2283640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:36.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba1a82640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:36.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba1a82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:36.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.453+0000 7f1ba1a82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:36.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2023-11-02T08:36:36.487 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 1 -- 172.21.15.151:0/2798847971 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0a8df0 msgr2=0x7f1b9c145580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:36.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0a8df0 0x7f1b9c145580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 1 -- 172.21.15.151:0/2798847971 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b9c0b13e0 msgr2=0x7f1b9c0ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:36.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b9c0b13e0 0x7f1b9c0ad5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 10 monclient: _finish_hunting 0 2023-11-02T08:36:36.488 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 1 monclient: found mon.noname-b 2023-11-02T08:36:36.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:36.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:36.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.456+0000 7f1ba2a84640 1 -- 172.21.15.151:0/2798847971 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b98009400 con 0x7f1b9c0b4170 2023-11-02T08:36:36.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.458+0000 7f1ba2a84640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b9c0b4170 0x7f1b9c1d96e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1b98034040 tx=0x7f1b980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:36.489 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.458+0000 7f1b8effd640 10 client.?.objecter ms_handle_connect 0x7f1b9c0b4170 2023-11-02T08:36:36.490 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.458+0000 7f1b8effd640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:36.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 1 -- 172.21.15.151:0/2798847971 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98032810 con 0x7f1b9c0b4170 2023-11-02T08:36:36.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:36.493 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:36.494 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 10 monclient: dump: 2023-11-02T08:36:36.494 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:36.494 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:36.495 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:36.495 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:36.495 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:36.496 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:36.496 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:36.496 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:36.496 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:36.496 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:36.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 10 monclient: _finish_auth 0 2023-11-02T08:36:36.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 10 monclient: _check_auth_tickets 2023-11-02T08:36:36.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:36.497 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 1 -- 172.21.15.151:0/2798847971 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b98032970 con 0x7f1b9c0b4170 2023-11-02T08:36:36.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:36.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 1 -- 172.21.15.151:0/2798847971 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98032d20 con 0x7f1b9c0b4170 2023-11-02T08:36:36.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba44e7640 5 monclient: authenticate success, global_id 4396 2023-11-02T08:36:36.498 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba44e7640 10 monclient: _renew_subs 2023-11-02T08:36:36.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba44e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:36.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9c0ae110 con 0x7f1b9c0b4170 2023-11-02T08:36:36.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba0a80640 4 set_mon_vals no callback set 2023-11-02T08:36:36.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:36.499 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:36.500 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1b8effd640 10 monclient: dump: 2023-11-02T08:36:36.500 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:36.500 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:36.500 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:36.500 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:36.501 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:36.501 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:36.501 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:36.501 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:36.502 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:36.502 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:36.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba44e7640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:36.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba44e7640 10 monclient: _renew_subs 2023-11-02T08:36:36.502 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba44e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:36.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9c0aced0 con 0x7f1b9c0b4170 2023-11-02T08:36:36.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.462+0000 7f1ba44e7640 1 librados: init done 2023-11-02T08:36:36.503 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.463+0000 7f1ba44e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:36.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.463+0000 7f1ba44e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:36.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.463+0000 7f1ba44e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:36.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.463+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9c0af590 con 0x7f1b9c0b4170 2023-11-02T08:36:36.504 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.481+0000 7f1b8effd640 1 -- 172.21.15.151:0/2798847971 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1b9804e440 con 0x7f1b9c0b4170 2023-11-02T08:36:36.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.481+0000 7f1b8effd640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f1b8006be40 0x7f1b8006e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:36.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.482+0000 7f1ba2283640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f1b8006be40 0x7f1b8006e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:36.505 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.482+0000 7f1ba2283640 10 monclient: get_auth_request con 0x7f1b8006be40 auth_method 0 2023-11-02T08:36:36.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.482+0000 7f1ba2283640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f1b8006be40 0x7f1b8006e300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1b94007fd0 tx=0x7f1b94007e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:36.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.482+0000 7f1b8effd640 10 client.4396.objecter ms_handle_connect 0x7f1b8006be40 2023-11-02T08:36:36.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.484+0000 7f1b8effd640 1 -- 172.21.15.151:0/2798847971 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1b98079cc0 con 0x7f1b9c0b4170 2023-11-02T08:36:36.506 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.484+0000 7f1b8effd640 10 client.4396.objecter ms_dispatch 0x7f1b9c000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:36.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.484+0000 7f1b8effd640 3 client.4396.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:36.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.484+0000 7f1b8effd640 3 client.4396.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:36.507 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.484+0000 7f1b8effd640 20 client.4396.objecter dump_active .. 0 homeless 2023-11-02T08:36:36.723 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.720+0000 7f1b8effd640 1 -- 172.21.15.151:0/2798847971 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f1b98076000 con 0x7f1b9c0b4170 2023-11-02T08:36:36.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.720+0000 7f1b8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:36.724 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.720+0000 7f1b8effd640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:36.886 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.883+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 --> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b9c0ade30 con 0x7f1b8006be40 2023-11-02T08:36:36.888 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.885+0000 7f1b8effd640 1 -- 172.21.15.151:0/2798847971 <== mgr.4108 v2:172.21.15.178:6800/85498 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48034 (secure 0 0 0) 0x7f1b9c0ade30 con 0x7f1b8006be40 2023-11-02T08:36:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:36:36.890 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2023-11-02T08:36:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 10 librados: watch_flush enter 2023-11-02T08:36:36.891 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 10 librados: watch_flush exit 2023-11-02T08:36:36.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 20 client.4396.objecter shutdown clearing up homeless session... 2023-11-02T08:36:36.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 10 client.4396.objecter successfully canceled tick 2023-11-02T08:36:36.892 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f1b8006be40 msgr2=0x7f1b8006e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:36.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f1b8006be40 0x7f1b8006e300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1b94007fd0 tx=0x7f1b94007e80 comp rx=0 tx=0).stop 2023-11-02T08:36:36.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 10 monclient: shutdown 2023-11-02T08:36:36.893 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b9c0b4170 msgr2=0x7f1b9c1d96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b9c0b4170 0x7f1b9c1d96e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1b98034040 tx=0x7f1b980046c0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 shutdown_connections 2023-11-02T08:36:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f1b8006be40 0x7f1b8006e300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.894 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b9c0b4170 0x7f1b9c1d96e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1b9c0b13e0 0x7f1b9c0ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 --2- 172.21.15.151:0/2798847971 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1b9c0a8df0 0x7f1b9c145580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.887+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 >> 172.21.15.151:0/2798847971 conn(0x7f1b9c065d80 msgr2=0x7f1b9c066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.888+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 shutdown_connections 2023-11-02T08:36:36.895 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.888+0000 7f1ba44e7640 1 -- 172.21.15.151:0/2798847971 wait complete. 2023-11-02T08:36:36.896 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.888+0000 7f1ba44e7640 1 librados: shutdown 2023-11-02T08:36:36.902 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-11-02T08:36:34.941234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":101,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218160,"kb_used_data":2544,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754756560,"statfs":{"total":773094113280,"available":772870717440,"internally_reserved":0,"allocated":2605056,"data_stored":1409152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":459280,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":77,"num_write":94,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.118940"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.443027+0000","last_change":"2023-11-02T08:36:29.169375+0000","last_active":"2023-11-02T08:36:30.443027+0000","last_peered":"2023-11-02T08:36:30.443027+0000","last_clean":"2023-11-02T08:36:30.443027+0000","last_became_active":"2023-11-02T08:36:29.139473+0000","last_became_peered":"2023-11-02T08:36:29.139473+0000","last_unstale":"2023-11-02T08:36:30.443027+0000","last_undegraded":"2023-11-02T08:36:30.443027+0000","last_fullsized":"2023-11-02T08:36:30.443027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T18:14:09.805168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.660528+0000","last_change":"2023-11-02T08:36:30.654204+0000","last_active":"2023-11-02T08:36:30.660528+0000","last_peered":"2023-11-02T08:36:30.660528+0000","last_clean":"2023-11-02T08:36:30.660528+0000","last_became_active":"2023-11-02T08:36:30.648476+0000","last_became_peered":"2023-11-02T08:36:30.648476+0000","last_unstale":"2023-11-02T08:36:30.660528+0000","last_undegraded":"2023-11-02T08:36:30.660528+0000","last_fullsized":"2023-11-02T08:36:30.660528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T14:11:58.899468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.926369+0000","last_change":"2023-11-02T08:36:28.981877+0000","last_active":"2023-11-02T08:36:30.926369+0000","last_peered":"2023-11-02T08:36:30.926369+0000","last_clean":"2023-11-02T08:36:30.926369+0000","last_became_active":"2023-11-02T08:36:28.976660+0000","last_became_peered":"2023-11-02T08:36:28.976660+0000","last_unstale":"2023-11-02T08:36:30.926369+0000","last_undegraded":"2023-11-02T08:36:30.926369+0000","last_fullsized":"2023-11-02T08:36:30.926369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T19:12:52.578679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.630882+0000","last_change":"2023-11-02T08:36:30.611750+0000","last_active":"2023-11-02T08:36:30.630882+0000","last_peered":"2023-11-02T08:36:30.630882+0000","last_clean":"2023-11-02T08:36:30.630882+0000","last_became_active":"2023-11-02T08:36:30.583185+0000","last_became_peered":"2023-11-02T08:36:30.583185+0000","last_unstale":"2023-11-02T08:36:30.630882+0000","last_undegraded":"2023-11-02T08:36:30.630882+0000","last_fullsized":"2023-11-02T08:36:30.630882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T14:18:01.847932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:31.105423+0000","last_change":"2023-11-02T08:36:29.148625+0000","last_active":"2023-11-02T08:36:31.105423+0000","last_peered":"2023-11-02T08:36:31.105423+0000","last_clean":"2023-11-02T08:36:31.105423+0000","last_became_active":"2023-11-02T08:36:29.121362+0000","last_became_peered":"2023-11-02T08:36:29.121362+0000","last_unstale":"2023-11-02T08:36:31.105423+0000","last_undegraded":"2023-11-02T08:36:31.105423+0000","last_fullsized":"2023-11-02T08:36:31.105423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T19:40:36.987496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.627752+0000","last_change":"2023-11-02T08:36:30.029990+0000","last_active":"2023-11-02T08:36:30.627752+0000","last_peered":"2023-11-02T08:36:30.627752+0000","last_clean":"2023-11-02T08:36:30.627752+0000","last_became_active":"2023-11-02T08:36:29.989223+0000","last_became_peered":"2023-11-02T08:36:29.989223+0000","last_unstale":"2023-11-02T08:36:30.627752+0000","last_undegraded":"2023-11-02T08:36:30.627752+0000","last_fullsized":"2023-11-02T08:36:30.627752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T16:23:35.917732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:30.918286+0000","last_change":"2023-11-02T08:36:29.095539+0000","last_active":"2023-11-02T08:36:30.918286+0000","last_peered":"2023-11-02T08:36:30.918286+0000","last_clean":"2023-11-02T08:36:30.918286+0000","last_became_active":"2023-11-02T08:36:29.090475+0000","last_became_peered":"2023-11-02T08:36:29.090475+0000","last_unstale":"2023-11-02T08:36:30.918286+0000","last_undegraded":"2023-11-02T08:36:30.918286+0000","last_fullsized":"2023-11-02T08:36:30.918286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T09:01:37.136232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-02T08:36:31.107475+0000","last_change":"2023-11-02T08:36:29.175364+0000","last_active":"2023-11-02T08:36:31.107475+0000","last_peered":"2023-11-02T08:36:31.107475+0000","last_clean":"2023-11-02T08:36:31.107475+0000","last_became_active":"2023-11-02T08:36:29.169756+0000","last_became_peered":"2023-11-02T08:36:29.169756+0000","last_unstale":"2023-11-02T08:36:31.107475+0000","last_undegraded":"2023-11-02T08:36:31.107475+0000","last_fullsized":"2023-11-02T08:36:31.107475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:27.509436+0000","last_clean_scrub_stamp":"2023-11-02T08:36:27.509436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T13:25:39.845560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'65","reported_seq":106,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-02T08:36:34.729115+0000","last_change":"2023-11-02T08:36:24.330685+0000","last_active":"2023-11-02T08:36:34.729115+0000","last_peered":"2023-11-02T08:36:34.729115+0000","last_clean":"2023-11-02T08:36:34.729115+0000","last_became_active":"2023-11-02T08:36:24.298799+0000","last_became_peered":"2023-11-02T08:36:24.298799+0000","last_unstale":"2023-11-02T08:36:34.729115+0000","last_undegraded":"2023-11-02T08:36:34.729115+0000","last_fullsized":"2023-11-02T08:36:34.729115+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-02T08:36:21.736705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-02T08:36:21.736705+0000","last_clean_scrub_stamp":"2023-11-02T08:36:21.736705+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-03T16:28:14.976849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":101,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":101,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":626688,"data_stored":474524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.476}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4560000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3460000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.967000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.382999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.757999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4710000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.98}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.587}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.105}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.148}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.047000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5259999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.162000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.273}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.832000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.936999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.516999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5869999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4930000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.195}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.756}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3719999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.18}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.34}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8420000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.776}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.818000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.343}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.404999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.654}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":39.683999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.058999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.189}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.817}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.838999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.667999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.739999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":225280,"data_stored":76684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1070000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7089999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.885999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.376000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":61.694000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.211}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.936}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":626688,"data_stored":474524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":65.715000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":151.25700000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0220000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.039999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":120.617}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":124.18899999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":32.896000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-02T08:36:36.903 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-02T08:36:36.903 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-02T08:36:36.904 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-02T08:36:36.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-02T08:36:37.002 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.999+0000 7f10539d0640 10 monclient: get_monmap_and_config 2023-11-02T08:36:37.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.999+0000 7f10539d0640 10 monclient: build_initial_monmap 2023-11-02T08:36:37.003 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:36.999+0000 7f10539d0640 10 monclient: monmap: 2023-11-02T08:36:37.003 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:37.003 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:36:37.000750+0000 2023-11-02T08:36:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:36:37.000750+0000 2023-11-02T08:36:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T08:36:37.005 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T08:36:37.005 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T08:36:37.005 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T08:36:37.005 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T08:36:37.005 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:37.007 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.000+0000 7f10539d0640 1 Processor -- start 2023-11-02T08:36:37.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.000+0000 7f10539d0640 1 -- start start 2023-11-02T08:36:37.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.000+0000 7f10539d0640 10 monclient: init 2023-11-02T08:36:37.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.000+0000 7f10539d0640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:37.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.000+0000 7f10539d0640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-11-02T08:36:37.008 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f104c0b4260 addr v1:172.21.15.178:6789/0 2023-11-02T08:36:37.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f104c141730 addr v1:172.21.15.151:6789/0 2023-11-02T08:36:37.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f104c14a230 0x7f104c14c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:37.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 10 monclient(hunting): picked mon.noname-c con 0x7f104c14a230 addr v2:172.21.15.178:3300/0 2023-11-02T08:36:37.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c0a9210 con 0x7f104c141730 2023-11-02T08:36:37.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c1422b0 con 0x7f104c0b4260 2023-11-02T08:36:37.009 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:37.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f104c0aa550 con 0x7f104c14a230 2023-11-02T08:36:37.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:37.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f10539d0640 10 monclient(hunting): authenticate will time out at 1368.356959s 2023-11-02T08:36:37.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f1051f6d640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f104c14a230 0x7f104c14c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:37.010 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f1050f6b640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f104c141730 0x7f104c141ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:34204/0 (socket says 172.21.15.151:34204) 2023-11-02T08:36:37.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.001+0000 7f1050f6b640 1 -- 172.21.15.151:0/1320921581 learned_addr learned my addr 172.21.15.151:0/1320921581 (peer_addr_for_me v1:172.21.15.151:0/0) 2023-11-02T08:36:37.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.002+0000 7f1051f6d640 10 monclient(hunting): get_auth_request con 0x7f104c14a230 auth_method 0 2023-11-02T08:36:37.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.002+0000 7f1051f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:37.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.002+0000 7f1051f6d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:37.011 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.002+0000 7f1051f6d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:37.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.002+0000 7f1051f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:37.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.002+0000 7f1051f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:37.012 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.002+0000 7f1051f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:37.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.005+0000 7f1051f6d640 10 monclient(hunting): handle_auth_done global_id 4399 payload 274 2023-11-02T08:36:37.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.005+0000 7f1051f6d640 1 -- 172.21.15.151:0/1320921581 >> v1:172.21.15.178:6789/0 conn(0x7f104c0b4260 legacy=0x7f104c0a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:37.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.005+0000 7f1051f6d640 1 -- 172.21.15.151:0/1320921581 >> v1:172.21.15.151:6789/0 conn(0x7f104c141730 legacy=0x7f104c141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:37.013 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.005+0000 7f1051f6d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:37.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.005+0000 7f1051f6d640 1 monclient: found mon.noname-c 2023-11-02T08:36:37.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.005+0000 7f1051f6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:37.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.005+0000 7f1051f6d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.178:3300/0 2023-11-02T08:36:37.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.005+0000 7f1051f6d640 1 -- 172.21.15.151:0/1320921581 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c0aa750 con 0x7f104c14a230 2023-11-02T08:36:37.014 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.009+0000 7f1051f6d640 1 --2- 172.21.15.151:0/1320921581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 0x7f104c14c620 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1044009dc0 tx=0x7f104402be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0d055d4b8274f7b server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:37.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.011+0000 7f1043fff640 1 -- 172.21.15.151:0/1320921581 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104402cc40 con 0x7f104c14a230 2023-11-02T08:36:37.015 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.011+0000 7f1043fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:37.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.011+0000 7f1043fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-11-02T08:36:37.016 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.011+0000 7f1043fff640 10 monclient: dump: 2023-11-02T08:36:37.016 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:37.016 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:37.016 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:37.017 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:37.017 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:37.017 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:37.017 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:37.017 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:37.017 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:37.018 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:37.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.011+0000 7f1043fff640 10 monclient: _finish_auth 0 2023-11-02T08:36:37.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.011+0000 7f1043fff640 10 monclient: _check_auth_tickets 2023-11-02T08:36:37.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.011+0000 7f1043fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:37.018 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.011+0000 7f10539d0640 5 monclient: authenticate success, global_id 4399 2023-11-02T08:36:37.019 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.011+0000 7f10539d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-11-02T08:36:37.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f1043fff640 1 -- 172.21.15.151:0/1320921581 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104402cda0 con 0x7f104c14a230 2023-11-02T08:36:37.021 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f1043fff640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:37.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f1043fff640 1 -- 172.21.15.151:0/1320921581 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044034660 con 0x7f104c14a230 2023-11-02T08:36:37.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f1043fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:37.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f1043fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:37.022 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f1043fff640 10 monclient: dump: 2023-11-02T08:36:37.022 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:37.023 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:37.023 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:37.023 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:37.023 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:37.023 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:37.024 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:37.024 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:37.024 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:37.024 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:37.024 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 10 monclient: get_monmap_and_config success 2023-11-02T08:36:37.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 4 set_mon_vals no callback set 2023-11-02T08:36:37.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 10 monclient: shutdown 2023-11-02T08:36:37.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:37.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 1 -- 172.21.15.151:0/1320921581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 msgr2=0x7f104c14c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:37.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 1 --2- 172.21.15.151:0/1320921581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 0x7f104c14c620 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1044009dc0 tx=0x7f104402be90 comp rx=0 tx=0).stop 2023-11-02T08:36:37.025 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 1 -- 172.21.15.151:0/1320921581 shutdown_connections 2023-11-02T08:36:37.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 1 --2- 172.21.15.151:0/1320921581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 0x7f104c14c620 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:37.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 1 -- 172.21.15.151:0/1320921581 >> 172.21.15.151:0/1320921581 conn(0x7f104c065d50 msgr2=0x7f104c066160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:37.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 1 -- 172.21.15.151:0/1320921581 shutdown_connections 2023-11-02T08:36:37.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.013+0000 7f10539d0640 1 -- 172.21.15.151:0/1320921581 wait complete. 2023-11-02T08:36:37.026 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.014+0000 7f10539d0640 10 monclient: build_initial_monmap 2023-11-02T08:36:37.027 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.014+0000 7f10539d0640 10 monclient: monmap: 2023-11-02T08:36:37.027 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T08:36:37.027 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:36:37.027 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2023-11-02T08:36:37.027 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2023-11-02T08:36:37.028 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T08:36:37.028 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:37.028 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:36:37.028 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:36:37.028 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:36:37.029 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:37.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.014+0000 7f10539d0640 1 librados: starting msgr at 2023-11-02T08:36:37.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.014+0000 7f10539d0640 1 librados: starting objecter 2023-11-02T08:36:37.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.014+0000 7f10539d0640 1 Processor -- start 2023-11-02T08:36:37.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.014+0000 7f10539d0640 1 -- start start 2023-11-02T08:36:37.029 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.014+0000 7f10539d0640 1 librados: setting wanted keys 2023-11-02T08:36:37.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.014+0000 7f10539d0640 1 librados: calling monclient init 2023-11-02T08:36:37.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.014+0000 7f10539d0640 10 monclient: init 2023-11-02T08:36:37.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient: _reopen_session rank -1 2023-11-02T08:36:37.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient: _add_conns ranks=[2,0,1] 2023-11-02T08:36:37.030 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f104c0b4260 0x7f104c12de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:37.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient(hunting): picked mon.noname-c con 0x7f104c0b4260 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:36:37.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f104c141730 0x7f104c13d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:37.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient(hunting): picked mon.noname-a con 0x7f104c141730 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:36:37.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 0x7f104c13dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:37.031 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient(hunting): picked mon.noname-b con 0x7f104c14a230 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:36:37.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:37.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c13cd10 con 0x7f104c141730 2023-11-02T08:36:37.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:37.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c13cf80 con 0x7f104c14a230 2023-11-02T08:36:37.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient(hunting): start opening mon connection 2023-11-02T08:36:37.032 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104c13d260 con 0x7f104c0b4260 2023-11-02T08:36:37.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient(hunting): _renew_subs 2023-11-02T08:36:37.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f10539d0640 10 monclient(hunting): authenticate will time out at 1368.371272s 2023-11-02T08:36:37.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.015+0000 7f1051f6d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 0x7f104c13dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:37.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f105176c640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f104c0b4260 0x7f104c12de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:37.033 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f1051f6d640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 0x7f104c13dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:36602/0 (socket says 172.21.15.151:36602) 2023-11-02T08:36:37.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f1051f6d640 1 -- 172.21.15.151:0/536493951 learned_addr learned my addr 172.21.15.151:0/536493951 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:36:37.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f1051f6d640 10 monclient(hunting): get_auth_request con 0x7f104c14a230 auth_method 0 2023-11-02T08:36:37.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f1051f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:37.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f1051f6d640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:37.034 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f1051f6d640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:37.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f105176c640 10 monclient(hunting): get_auth_request con 0x7f104c0b4260 auth_method 0 2023-11-02T08:36:37.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f105176c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:36:37.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f105176c640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:36:37.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f105176c640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:36:37.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f1051f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:37.035 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f1051f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:37.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f1051f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:37.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f105176c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:36:37.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f105176c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:36:37.036 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.016+0000 7f105176c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:36:37.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2023-11-02T08:36:37.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 1 -- 172.21.15.151:0/536493951 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f104c0b4260 msgr2=0x7f104c12de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:37.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f104c0b4260 0x7f104c12de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:37.038 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 1 -- 172.21.15.151:0/536493951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f104c141730 msgr2=0x7f104c13d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:37.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f104c141730 0x7f104c13d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:37.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 10 monclient: _finish_hunting 0 2023-11-02T08:36:37.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 1 monclient: found mon.noname-b 2023-11-02T08:36:37.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:36:37.039 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:36:37.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.019+0000 7f1051f6d640 1 -- 172.21.15.151:0/536493951 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044009a70 con 0x7f104c14a230 2023-11-02T08:36:37.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.021+0000 7f1051f6d640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 0x7f104c13dea0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1044009ee0 tx=0x7f1044004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:37.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.021+0000 7f1041ffb640 10 client.?.objecter ms_handle_connect 0x7f104c14a230 2023-11-02T08:36:37.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.021+0000 7f1041ffb640 10 client.?.objecter resend_mon_ops 2023-11-02T08:36:37.040 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 1 -- 172.21.15.151:0/536493951 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104403e610 con 0x7f104c14a230 2023-11-02T08:36:37.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:37.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:36:37.041 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 10 monclient: dump: 2023-11-02T08:36:37.041 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:37.041 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:37.041 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:37.042 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:37.042 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:37.042 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:37.042 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:37.042 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:37.043 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:37.043 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:37.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 10 monclient: _finish_auth 0 2023-11-02T08:36:37.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 10 monclient: _check_auth_tickets 2023-11-02T08:36:37.043 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:36:37.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 1 -- 172.21.15.151:0/536493951 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104403e770 con 0x7f104c14a230 2023-11-02T08:36:37.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:36:37.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 1 -- 172.21.15.151:0/536493951 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104403eae0 con 0x7f104c14a230 2023-11-02T08:36:37.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:36:37.044 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:36:37.045 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f1041ffb640 10 monclient: dump: 2023-11-02T08:36:37.045 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2023-11-02T08:36:37.045 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:36:37.045 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:37.045 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:36:37.045 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 18 (reef) 2023-11-02T08:36:37.046 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T08:36:37.046 INFO:teuthology.orchestra.run.smithi151.stderr:0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:36:37.046 INFO:teuthology.orchestra.run.smithi151.stderr:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:36:37.046 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:36:37.047 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T08:36:37.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.025+0000 7f10539d0640 5 monclient: authenticate success, global_id 4405 2023-11-02T08:36:37.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.025+0000 7f10539d0640 10 monclient: _renew_subs 2023-11-02T08:36:37.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.025+0000 7f10539d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:37.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.025+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104c13d540 con 0x7f104c14a230 2023-11-02T08:36:37.047 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.024+0000 7f10437fe640 4 set_mon_vals no callback set 2023-11-02T08:36:37.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.025+0000 7f10539d0640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:36:37.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.025+0000 7f10539d0640 10 monclient: _renew_subs 2023-11-02T08:36:37.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.025+0000 7f10539d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:37.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.025+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c14a9f0 con 0x7f104c14a230 2023-11-02T08:36:37.048 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.025+0000 7f10539d0640 1 librados: init done 2023-11-02T08:36:37.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.026+0000 7f10539d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:37.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.026+0000 7f10539d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:37.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.026+0000 7f10539d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:37.049 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.026+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104c14c390 con 0x7f104c14a230 2023-11-02T08:36:37.270 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.266+0000 7f1041ffb640 1 -- 172.21.15.151:0/536493951 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f104403b050 con 0x7f104c14a230 2023-11-02T08:36:37.270 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.267+0000 7f1041ffb640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f103006bdc0 0x7f103006e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:36:37.271 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.267+0000 7f105176c640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f103006bdc0 0x7f103006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:36:37.271 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.267+0000 7f105176c640 10 monclient: get_auth_request con 0x7f103006bdc0 auth_method 0 2023-11-02T08:36:37.271 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.268+0000 7f1041ffb640 10 client.4405.objecter ms_handle_connect 0x7f103006bdc0 2023-11-02T08:36:37.271 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.268+0000 7f105176c640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f103006bdc0 0x7f103006e280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1048000f60 tx=0x7f1048009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:36:37.272 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.269+0000 7f1041ffb640 1 -- 172.21.15.151:0/536493951 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f104407a970 con 0x7f104c14a230 2023-11-02T08:36:37.272 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.269+0000 7f1041ffb640 10 client.4405.objecter ms_dispatch 0x7f104c000b90 osd_map(22..22 src has 1..22) v4 2023-11-02T08:36:37.272 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.269+0000 7f1041ffb640 3 client.4405.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:36:37.273 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.269+0000 7f1041ffb640 3 client.4405.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:36:37.274 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.269+0000 7f1041ffb640 20 client.4405.objecter dump_active .. 0 homeless 2023-11-02T08:36:37.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.279+0000 7f1041ffb640 1 -- 172.21.15.151:0/536493951 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179298 (secure 0 0 0) 0x7f1044076cb0 con 0x7f104c14a230 2023-11-02T08:36:37.282 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.279+0000 7f1041ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-11-02T08:36:37.283 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.279+0000 7f1041ffb640 10 monclient: _finish_command 1 = system:0 2023-11-02T08:36:37.586 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.583+0000 7f10539d0640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-11-02T08:36:37.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.583+0000 7f10539d0640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-11-02T08:36:37.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.583+0000 7f10539d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.178:3300/0 2023-11-02T08:36:37.587 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.583+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f104c14c390 con 0x7f104c14a230 2023-11-02T08:36:37.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.589+0000 7f1041ffb640 1 -- 172.21.15.151:0/536493951 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f104c14c390 con 0x7f104c14a230 2023-11-02T08:36:37.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.589+0000 7f1041ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-11-02T08:36:37.595 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.589+0000 7f1041ffb640 10 monclient: _finish_command 2 = system:0 2023-11-02T08:36:37.595 INFO:teuthology.orchestra.run.smithi151.stdout: 2023-11-02T08:36:37.596 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-02T08:36:37.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 10 librados: watch_flush enter 2023-11-02T08:36:37.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 10 librados: watch_flush exit 2023-11-02T08:36:37.597 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 20 client.4405.objecter shutdown clearing up homeless session... 2023-11-02T08:36:37.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 10 client.4405.objecter successfully canceled tick 2023-11-02T08:36:37.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f103006bdc0 msgr2=0x7f103006e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:37.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f103006bdc0 0x7f103006e280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1048000f60 tx=0x7f1048009040 comp rx=0 tx=0).stop 2023-11-02T08:36:37.598 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 10 monclient: shutdown 2023-11-02T08:36:37.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:36:37.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 msgr2=0x7f104c13dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:36:37.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 0x7f104c13dea0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1044009ee0 tx=0x7f1044004a40 comp rx=0 tx=0).stop 2023-11-02T08:36:37.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 shutdown_connections 2023-11-02T08:36:37.599 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7f103006bdc0 0x7f103006e280 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:37.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f104c14a230 0x7f104c13dea0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:37.600 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f104c141730 0x7f104c13d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:37.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 --2- 172.21.15.151:0/536493951 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f104c0b4260 0x7f104c12de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:36:37.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 >> 172.21.15.151:0/536493951 conn(0x7f104c065d50 msgr2=0x7f104c0b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:36:37.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 shutdown_connections 2023-11-02T08:36:37.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 -- 172.21.15.151:0/536493951 wait complete. 2023-11-02T08:36:37.601 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T08:36:37.591+0000 7f10539d0640 1 librados: shutdown 2023-11-02T08:36:37.607 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-02T08:36:37.608 INFO:teuthology.run_tasks:Running task workunit... 2023-11-02T08:36:37.620 INFO:tasks.workunit:Pulling workunits from ref b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:36:37.621 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-02T08:36:37.621 DEBUG:teuthology.orchestra.run.smithi151:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-02T08:36:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T08:36:37.637 INFO:teuthology.orchestra.run.smithi151.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-11-02T08:36:37.637 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-11-02T08:36:37.691 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-11-02T08:36:37.692 DEBUG:teuthology.orchestra.run.smithi151:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-11-02T08:36:37.746 INFO:tasks.workunit:timeout=6h 2023-11-02T08:36:37.747 INFO:tasks.workunit:cleanup=True 2023-11-02T08:36:37.747 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:36:37.803 INFO:tasks.workunit.client.0.smithi151.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-02T08:36:49.236 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:01:30.126 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:49.239 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:01:30.129 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:49.278 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:01:30.169 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:49.282 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:30.158 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:49.291 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:30.166 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:49.306 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:01:30.196 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:49.312 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:01:30.202 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:49.328 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:30.204 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:36:49.376 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:01:30.266 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:09.352 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:01:50.243 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:09.355 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:01:50.246 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:09.380 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:01:50.271 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:09.402 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:50.277 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:09.405 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:50.280 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:09.419 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:50.294 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:09.436 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:01:50.327 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:09.442 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:01:50.333 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:09.465 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:01:50.355 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:24.940 INFO:tasks.workunit.client.0.smithi151.stderr:Note: switching to 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081'. 2023-11-02T08:37:24.941 INFO:tasks.workunit.client.0.smithi151.stderr: 2023-11-02T08:37:24.941 INFO:tasks.workunit.client.0.smithi151.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-02T08:37:24.941 INFO:tasks.workunit.client.0.smithi151.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-02T08:37:24.942 INFO:tasks.workunit.client.0.smithi151.stderr:state without impacting any branches by switching back to a branch. 2023-11-02T08:37:24.942 INFO:tasks.workunit.client.0.smithi151.stderr: 2023-11-02T08:37:24.942 INFO:tasks.workunit.client.0.smithi151.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-02T08:37:24.943 INFO:tasks.workunit.client.0.smithi151.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-02T08:37:24.943 INFO:tasks.workunit.client.0.smithi151.stderr: 2023-11-02T08:37:24.944 INFO:tasks.workunit.client.0.smithi151.stderr: git switch -c 2023-11-02T08:37:24.944 INFO:tasks.workunit.client.0.smithi151.stderr: 2023-11-02T08:37:24.944 INFO:tasks.workunit.client.0.smithi151.stderr:Or undo this operation with: 2023-11-02T08:37:24.944 INFO:tasks.workunit.client.0.smithi151.stderr: 2023-11-02T08:37:24.944 INFO:tasks.workunit.client.0.smithi151.stderr: git switch - 2023-11-02T08:37:24.945 INFO:tasks.workunit.client.0.smithi151.stderr: 2023-11-02T08:37:24.945 INFO:tasks.workunit.client.0.smithi151.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-02T08:37:24.945 INFO:tasks.workunit.client.0.smithi151.stderr: 2023-11-02T08:37:24.945 INFO:tasks.workunit.client.0.smithi151.stderr:HEAD is now at b5a9486b4a6 Merge branch 'wip-63387-reef' of https://github.com/ajarr/ceph into wip-yuri6-testing-2023-11-01-0745-reef 2023-11-02T08:37:24.947 DEBUG:teuthology.orchestra.run.smithi151:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-02T08:37:25.011 INFO:tasks.workunit.client.0.smithi151.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-02T08:37:25.012 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-02T08:37:25.013 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-02T08:37:25.634 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-02T08:37:25.694 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-02T08:37:25.733 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-02T08:37:25.735 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-02T08:37:25.735 INFO:tasks.workunit.client.0.smithi151.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-02T08:37:25.774 INFO:tasks.workunit.client.0.smithi151.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-02T08:37:25.777 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2023-11-02T08:37:25.777 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-02T08:37:25.832 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-11-02T08:37:25.833 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-11-02T08:37:25.834 DEBUG:teuthology.orchestra.run.smithi151:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-11-02T08:37:25.892 INFO:tasks.workunit.client.0.smithi151.stderr:+ parallel=1 2023-11-02T08:37:25.892 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' '' = --serial ']' 2023-11-02T08:37:25.892 INFO:tasks.workunit.client.0.smithi151.stderr:+ color= 2023-11-02T08:37:25.892 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -t 1 ']' 2023-11-02T08:37:25.893 INFO:tasks.workunit.client.0.smithi151.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-11-02T08:37:25.893 INFO:tasks.workunit.client.0.smithi151.stderr:+ declare -A pids 2023-11-02T08:37:25.893 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:25.893 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:25.894 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_aio 2023-11-02T08:37:25.894 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_aio' 2023-11-02T08:37:25.894 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_aio 2023-11-02T08:37:25.894 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:25.895 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_aio 2023-11-02T08:37:25.895 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91289 2023-11-02T08:37:25.896 INFO:tasks.workunit.client.0.smithi151.stdout:test api_aio on pid 91289 2023-11-02T08:37:25.899 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_aio on pid 91289' 2023-11-02T08:37:25.899 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91289 2023-11-02T08:37:25.899 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:25.899 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:25.900 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-11-02T08:37:25.900 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_aio_pp 2023-11-02T08:37:25.900 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_aio_pp' 2023-11-02T08:37:25.900 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_aio_pp 2023-11-02T08:37:25.900 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:25.901 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:25.901 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:25.901 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:25.901 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:25.901 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.902 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:25.902 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:25.902 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:25.902 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:25.902 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.903 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:25.903 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.903 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:25.903 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:25.903 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.904 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:25.904 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.904 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:25.904 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:25.905 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:25.905 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.905 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:25.905 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.905 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:25.906 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:25.906 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_aio_pp 2023-11-02T08:37:25.906 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91296 2023-11-02T08:37:25.907 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_aio_pp on pid 91296' 2023-11-02T08:37:25.907 INFO:tasks.workunit.client.0.smithi151.stdout:test api_aio_pp on pid 91296 2023-11-02T08:37:25.913 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91296 2023-11-02T08:37:25.913 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:25.913 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:25.913 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-11-02T08:37:25.914 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_io 2023-11-02T08:37:25.914 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_io' 2023-11-02T08:37:25.914 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_io 2023-11-02T08:37:25.915 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:25.915 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:25.915 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:25.915 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:25.915 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:25.915 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.916 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:25.916 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:25.916 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:25.916 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:25.916 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:25.917 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:25.917 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:25.917 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.917 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.917 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:25.918 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:25.918 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.918 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.918 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:25.918 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:25.919 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:25.919 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:25.919 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.919 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:25.919 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.920 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:25.920 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:25.920 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.920 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:25.920 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:25.920 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:25.921 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.921 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.921 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:25.921 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:25.921 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.922 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:25.922 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:25.922 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:25.922 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:25.922 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.923 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:25.923 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.923 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:25.923 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:25.923 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_io 2023-11-02T08:37:25.923 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91305 2023-11-02T08:37:25.924 INFO:tasks.workunit.client.0.smithi151.stdout:test api_io on pid 91305 2023-11-02T08:37:25.928 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_io on pid 91305' 2023-11-02T08:37:25.929 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91305 2023-11-02T08:37:25.929 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:25.929 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:25.929 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-11-02T08:37:25.930 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_io_pp 2023-11-02T08:37:25.930 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_io_pp' 2023-11-02T08:37:25.930 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_io_pp 2023-11-02T08:37:25.930 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:25.931 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:25.931 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:25.931 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:25.931 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:25.932 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.932 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:25.932 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:25.932 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:25.933 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:25.937 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:25.937 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:25.937 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:25.938 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:25.938 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.938 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:25.939 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.939 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:25.939 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.939 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:25.940 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.940 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:25.940 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:25.940 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:25.940 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:25.941 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:25.941 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:25.941 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:25.941 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.942 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:25.942 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.942 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:25.942 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:25.943 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:25.943 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.943 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.943 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:25.943 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:25.944 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.944 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:25.944 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.944 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:25.945 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:25.945 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:25.945 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.947 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:25.947 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.947 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:25.947 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:25.948 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:25.950 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.950 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:25.951 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.954 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:25.954 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:25.955 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_io_pp 2023-11-02T08:37:25.955 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91316 2023-11-02T08:37:25.956 INFO:tasks.workunit.client.0.smithi151.stdout:test api_io_pp on pid 91316 2023-11-02T08:37:25.967 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_io_pp on pid 91316' 2023-11-02T08:37:25.968 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91316 2023-11-02T08:37:25.968 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-11-02T08:37:25.968 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:25.969 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:25.969 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_asio 2023-11-02T08:37:25.969 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_asio' 2023-11-02T08:37:25.969 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_asio 2023-11-02T08:37:25.970 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:25.970 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:25.970 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:25.970 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:25.970 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.971 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:25.971 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:25.971 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.971 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:25.972 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:25.972 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:25.972 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.973 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:25.973 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.973 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:25.973 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:25.973 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:25.978 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.979 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:25.979 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.980 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:25.980 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:25.980 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.981 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:25.981 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:25.981 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:25.981 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:25.981 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.982 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:25.982 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.982 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:25.982 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:25.982 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:25.983 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:25.983 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:25.983 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:25.983 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:25.984 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.984 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:25.984 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.984 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:25.984 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:25.985 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:25.985 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:25.985 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:25.985 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.985 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:25.985 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.986 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.986 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:25.986 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:25.986 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:25.986 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.987 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:25.987 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:25.987 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.987 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:25.987 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.988 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:25.988 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:25.988 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:25.988 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:25.988 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.988 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:25.989 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.989 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:25.989 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:25.989 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:25.989 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:25.990 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:25.990 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:25.990 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:25.990 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:25.990 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:25.991 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:25.991 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_asio 2023-11-02T08:37:25.991 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:25.991 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:25.992 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:25.992 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91330 2023-11-02T08:37:25.992 INFO:tasks.workunit.client.0.smithi151.stdout:test api_asio on pid 91330 2023-11-02T08:37:26.000 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_asio on pid 91330' 2023-11-02T08:37:26.000 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91330 2023-11-02T08:37:26.000 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.000 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.001 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-11-02T08:37:26.001 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.001 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_list 2023-11-02T08:37:26.001 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_list' 2023-11-02T08:37:26.001 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_list 2023-11-02T08:37:26.002 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.002 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.002 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.003 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.003 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.003 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.003 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.003 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.003 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.004 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.004 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.004 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.004 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.004 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.005 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.005 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.005 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.005 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.006 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.006 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.006 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.006 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.007 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.007 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.007 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.007 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.007 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.008 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.008 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.008 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.008 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.008 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.009 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.009 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.009 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.009 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.009 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.010 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.011 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.011 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.011 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.012 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.012 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.012 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.012 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.012 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.013 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.013 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.013 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.013 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.013 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.014 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.014 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.014 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.014 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.015 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.015 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.015 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.015 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.015 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.016 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.016 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.016 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.016 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.016 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.017 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.017 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.020 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.021 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.021 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.021 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.021 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.022 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.022 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.022 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.022 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.023 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.023 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.023 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.023 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.024 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.024 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91289/exe 2023-11-02T08:37:26.024 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.024 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.025 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.025 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.025 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.025 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.025 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.026 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.026 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.026 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.026 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.026 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.027 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.027 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.027 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.027 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.027 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.028 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.028 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.028 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_list 2023-11-02T08:37:26.028 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.028 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91350 2023-11-02T08:37:26.029 INFO:tasks.workunit.client.0.smithi151.stdout:test api_list on pid 91350 2023-11-02T08:37:26.036 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_list on pid 91350' 2023-11-02T08:37:26.037 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91350 2023-11-02T08:37:26.037 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.037 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.037 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-11-02T08:37:26.038 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_lock 2023-11-02T08:37:26.038 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.038 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.038 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.039 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.039 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.039 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_lock' 2023-11-02T08:37:26.039 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.039 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.039 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.040 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.040 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.040 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_lock 2023-11-02T08:37:26.040 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.040 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.041 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.041 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.041 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_aio 2023-11-02T08:37:26.041 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.041 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.042 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.042 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.042 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.042 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.042 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.043 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.043 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.043 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.043 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.043 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.044 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.044 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.044 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.045 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.045 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.045 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.045 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.045 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.046 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.046 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.046 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.046 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.046 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_aio: /' 2023-11-02T08:37:26.047 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.047 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.047 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.047 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.047 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.048 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.048 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.048 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.048 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.048 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.049 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.049 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.049 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.049 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.049 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.050 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.050 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.050 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.050 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.050 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.051 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.051 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.052 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.052 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.052 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.052 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.053 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.053 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.053 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.053 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.053 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.054 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.054 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.054 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.054 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.054 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.055 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.055 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.055 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.056 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.056 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.056 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.056 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.056 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.057 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.057 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.057 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_aio.log 2023-11-02T08:37:26.057 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.057 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91296/exe 2023-11-02T08:37:26.058 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.058 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.058 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.058 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.058 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.059 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.059 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.059 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.059 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.060 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.060 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.060 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.060 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.060 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.061 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.061 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.061 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.061 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.062 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.062 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.062 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.062 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_lock 2023-11-02T08:37:26.062 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.063 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.063 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.063 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.063 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.063 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91369 2023-11-02T08:37:26.064 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_lock on pid 91369' 2023-11-02T08:37:26.064 INFO:tasks.workunit.client.0.smithi151.stdout:test api_lock on pid 91369 2023-11-02T08:37:26.074 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-11-02T08:37:26.074 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91369 2023-11-02T08:37:26.074 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.074 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.075 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.075 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.075 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.075 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.076 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.076 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.076 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_lock_pp 2023-11-02T08:37:26.076 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.079 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.080 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.080 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.080 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.080 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.080 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.081 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.081 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.082 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.082 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_aio_pp 2023-11-02T08:37:26.082 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.082 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.083 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.083 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.083 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.083 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.083 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.084 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.084 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.084 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.084 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-11-02T08:37:26.084 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.086 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.087 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.087 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.087 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.087 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.087 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.088 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.088 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.088 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.088 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.088 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.089 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.090 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.091 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.091 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.091 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.092 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.092 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.092 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.093 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.093 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.093 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.093 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_lock_pp' 2023-11-02T08:37:26.094 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.094 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.094 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.095 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.095 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.095 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.095 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.096 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.096 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.096 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.097 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.097 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.097 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_aio_pp: /' 2023-11-02T08:37:26.098 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.099 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.100 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.100 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.100 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.100 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.100 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_lock_pp 2023-11-02T08:37:26.101 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.101 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.101 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.101 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91305/exe 2023-11-02T08:37:26.101 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.101 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.102 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.102 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.102 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.102 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.102 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.103 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.103 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.103 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.103 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.103 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.104 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.104 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.104 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.104 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.104 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.104 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.105 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.105 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.105 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.105 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.105 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.106 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.106 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.106 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.106 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.106 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.107 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.107 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.107 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.107 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.107 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.108 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.109 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.109 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_lock_pp 2023-11-02T08:37:26.109 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91389 2023-11-02T08:37:26.109 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_lock_pp on pid 91389' 2023-11-02T08:37:26.110 INFO:tasks.workunit.client.0.smithi151.stdout:test api_lock_pp on pid 91389 2023-11-02T08:37:26.120 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91389 2023-11-02T08:37:26.121 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.121 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.121 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.121 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.122 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.122 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.122 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.124 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.124 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.124 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.125 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_misc 2023-11-02T08:37:26.125 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.125 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.125 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.125 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_misc' 2023-11-02T08:37:26.126 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.126 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.126 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-11-02T08:37:26.126 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.127 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.127 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.127 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.127 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.127 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.128 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.128 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.128 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.128 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.128 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.129 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.129 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.129 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.129 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.130 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.130 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.130 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.130 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.130 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.131 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.131 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.131 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.131 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.131 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.132 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.132 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.132 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.132 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.132 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.133 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.133 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.133 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.134 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.135 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.135 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.135 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.136 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.136 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.136 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.136 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_misc 2023-11-02T08:37:26.137 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.137 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.137 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.137 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.137 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.138 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.138 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.138 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.138 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.138 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.139 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_io.log 2023-11-02T08:37:26.139 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.139 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.139 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.139 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.139 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.140 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.140 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_io: /' 2023-11-02T08:37:26.140 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.140 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.140 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.141 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.141 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.141 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.141 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.141 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.142 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.142 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.142 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.142 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.142 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.143 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.143 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.143 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.143 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.143 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.144 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.144 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.144 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.144 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.145 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_io 2023-11-02T08:37:26.145 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.145 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.145 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.145 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.145 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.146 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.146 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.146 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.146 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.146 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.147 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.147 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.147 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.147 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.147 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.148 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.148 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.148 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.148 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.148 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.149 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.149 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91316/exe 2023-11-02T08:37:26.149 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.149 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.149 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.150 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.150 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.150 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.150 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.150 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.151 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.151 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.151 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.151 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.151 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.152 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.152 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.152 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.152 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_misc 2023-11-02T08:37:26.152 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.153 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.153 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.153 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91410 2023-11-02T08:37:26.153 INFO:tasks.workunit.client.0.smithi151.stdout:test api_misc on pid 91410 2023-11-02T08:37:26.164 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_misc on pid 91410' 2023-11-02T08:37:26.164 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91410 2023-11-02T08:37:26.164 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.164 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.165 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.165 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.165 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-11-02T08:37:26.165 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.166 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.166 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_misc_pp 2023-11-02T08:37:26.166 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.166 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.166 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.167 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.167 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.167 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.167 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.167 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.168 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.168 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.168 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.168 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.168 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.169 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.169 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.169 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.169 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_misc_pp' 2023-11-02T08:37:26.169 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.170 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.170 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.170 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.170 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.170 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.171 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.171 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.171 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.171 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.171 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.172 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.172 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.172 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.172 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.172 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.173 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.173 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.173 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.173 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.174 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.174 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_io_pp: /' 2023-11-02T08:37:26.174 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.175 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_io_pp 2023-11-02T08:37:26.175 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.175 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.175 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91330/exe 2023-11-02T08:37:26.175 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.176 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.176 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.176 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.176 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.176 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.177 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.177 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.177 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.177 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.177 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.178 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.178 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.178 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.178 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.178 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.179 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_misc_pp 2023-11-02T08:37:26.179 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.179 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.179 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.180 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.180 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.180 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.181 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.181 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.181 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.181 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.181 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.181 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.182 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.182 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.182 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.182 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.182 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.183 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.183 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.183 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.183 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.183 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.184 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.184 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.184 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.184 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.184 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.185 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.185 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.185 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.185 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.185 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.185 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.186 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.186 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.186 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.186 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.187 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.187 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.187 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.187 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.187 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.188 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.188 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.188 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.188 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.188 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_asio 2023-11-02T08:37:26.189 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_asio.log 2023-11-02T08:37:26.189 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_misc_pp 2023-11-02T08:37:26.189 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_asio: /' 2023-11-02T08:37:26.189 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91432 2023-11-02T08:37:26.189 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_misc_pp on pid 91432' 2023-11-02T08:37:26.190 INFO:tasks.workunit.client.0.smithi151.stdout:test api_misc_pp on pid 91432 2023-11-02T08:37:26.197 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91432 2023-11-02T08:37:26.198 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.198 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.198 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-11-02T08:37:26.198 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.198 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.199 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.199 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.199 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.199 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.199 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.200 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.200 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.200 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_tier_pp 2023-11-02T08:37:26.200 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.200 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.201 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_tier_pp' 2023-11-02T08:37:26.201 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.201 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.201 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.201 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.202 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.202 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.202 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.202 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.203 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.203 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.203 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.203 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.203 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.203 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.204 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.204 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.204 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.204 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.204 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.205 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.205 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.205 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.205 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.205 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.206 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.206 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.206 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.206 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.207 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.207 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.207 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.207 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.207 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.207 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_tier_pp 2023-11-02T08:37:26.208 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.208 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.208 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.208 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-11-02T08:37:26.208 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.209 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.209 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.209 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.209 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.209 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.210 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.210 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.210 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.210 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.211 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.211 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.211 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.211 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.211 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.212 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.212 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.212 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.212 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.212 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.213 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.213 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.213 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.213 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.213 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.214 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.214 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_tier_pp 2023-11-02T08:37:26.214 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91447 2023-11-02T08:37:26.214 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_tier_pp on pid 91447' 2023-11-02T08:37:26.215 INFO:tasks.workunit.client.0.smithi151.stdout:test api_tier_pp on pid 91447 2023-11-02T08:37:26.235 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91447 2023-11-02T08:37:26.236 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.236 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.236 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-11-02T08:37:26.236 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.236 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.237 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.237 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.237 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.237 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_pool 2023-11-02T08:37:26.238 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.238 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.238 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.238 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.238 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.239 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.239 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.239 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.240 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.240 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.240 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.240 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.240 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.241 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.241 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.241 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.241 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.241 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.242 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.242 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.242 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.242 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.242 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.243 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.243 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.243 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.243 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.244 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.244 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.244 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.244 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.244 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.245 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.245 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.245 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.245 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.245 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_pool' 2023-11-02T08:37:26.246 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.246 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.246 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.246 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.246 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.247 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.247 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.247 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_pool 2023-11-02T08:37:26.247 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.248 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.248 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.248 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.248 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.248 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.249 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.249 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.249 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.249 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.249 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.250 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.250 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.250 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.250 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.250 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.251 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.251 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.251 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.251 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.251 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.252 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.252 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.252 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.252 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.252 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.253 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.253 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.253 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91369/exe 2023-11-02T08:37:26.253 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.253 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.254 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.254 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.254 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.254 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.254 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.255 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.255 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.255 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.255 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.255 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.256 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.256 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.256 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.256 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.256 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.257 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.257 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.257 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.257 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.257 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.257 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.258 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.258 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.258 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.258 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.258 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.259 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.259 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.259 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.259 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.260 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.260 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.260 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.260 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.260 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.260 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.261 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.261 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.261 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.261 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_lock: /' 2023-11-02T08:37:26.261 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.262 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.262 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.262 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.262 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.262 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.263 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_lock 2023-11-02T08:37:26.263 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.263 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.263 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.263 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.264 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.264 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.264 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.264 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.264 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.264 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.265 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.265 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.265 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.265 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.265 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.266 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.266 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91350/exe 2023-11-02T08:37:26.266 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.266 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.267 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.267 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.267 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.267 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.267 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.268 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.268 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.268 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.268 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.268 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.269 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.269 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_lock.log 2023-11-02T08:37:26.269 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.269 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.269 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.270 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.270 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.270 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.270 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.270 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.271 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.271 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.271 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.271 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.271 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.272 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.272 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.272 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.272 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.272 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.273 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.273 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.273 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.273 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.274 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_list.log 2023-11-02T08:37:26.274 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.274 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.274 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.274 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.275 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.275 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.275 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_list 2023-11-02T08:37:26.275 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.275 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.276 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.276 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.276 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.276 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.276 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.277 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.277 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.277 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.277 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.277 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.278 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.278 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.278 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.278 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.278 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.279 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.279 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.279 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.279 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.279 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.280 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.280 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.280 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.280 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.280 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.280 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.281 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.281 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.281 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.281 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.281 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.282 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.282 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.282 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.282 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.282 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.283 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.283 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.283 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.283 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.283 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.284 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.284 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.284 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.284 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.284 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.285 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.285 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.285 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.285 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.285 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.285 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.286 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_list: /' 2023-11-02T08:37:26.286 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.286 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.286 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.286 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.287 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.287 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.287 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.287 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.287 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.288 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.288 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.288 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.288 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.288 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.289 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.289 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.289 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.289 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.289 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.290 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.290 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.290 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.290 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.291 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.291 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.291 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.291 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.291 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.292 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.292 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.292 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.292 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.292 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.293 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.293 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.293 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_pool 2023-11-02T08:37:26.293 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91500 2023-11-02T08:37:26.294 INFO:tasks.workunit.client.0.smithi151.stdout:test api_pool on pid 91500 2023-11-02T08:37:26.305 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_pool on pid 91500' 2023-11-02T08:37:26.305 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91500 2023-11-02T08:37:26.305 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.306 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.306 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-11-02T08:37:26.306 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.307 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.307 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.307 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.307 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.307 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.307 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.308 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.308 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_snapshots 2023-11-02T08:37:26.308 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.308 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.308 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.309 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.309 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.309 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.309 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.309 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.310 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.310 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.310 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_snapshots' 2023-11-02T08:37:26.310 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.310 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.311 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.311 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.311 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.311 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.311 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.312 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.312 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.312 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.312 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91389/exe 2023-11-02T08:37:26.313 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.313 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.313 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91410/exe 2023-11-02T08:37:26.313 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.313 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.314 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.314 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.314 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.314 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.314 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.315 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.315 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.315 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.315 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.315 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.316 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.316 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.316 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.316 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.316 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.317 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.317 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.317 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.317 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.317 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.317 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.318 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.318 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.318 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.318 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.318 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.319 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.319 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.319 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.319 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.319 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.320 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.320 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.320 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.320 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.320 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.321 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_misc: /' 2023-11-02T08:37:26.321 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.321 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.321 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_misc 2023-11-02T08:37:26.322 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.322 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.322 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.322 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.322 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.322 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.323 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.323 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.323 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.323 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.323 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.324 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.324 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.324 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.324 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.324 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.325 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.325 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.325 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_misc.log 2023-11-02T08:37:26.325 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-11-02T08:37:26.325 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.326 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.326 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.326 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.327 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.327 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.327 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.327 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.327 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.328 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91432/exe 2023-11-02T08:37:26.328 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.328 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.328 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.328 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.329 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.329 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.329 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.329 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.329 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.330 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.330 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_lock_pp 2023-11-02T08:37:26.330 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.330 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_lock_pp: /' 2023-11-02T08:37:26.330 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.331 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.331 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.331 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.331 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.331 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.331 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.332 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.332 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.332 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.332 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_snapshots 2023-11-02T08:37:26.332 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-11-02T08:37:26.333 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_snapshots 2023-11-02T08:37:26.333 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91545 2023-11-02T08:37:26.333 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_snapshots on pid 91545' 2023-11-02T08:37:26.334 INFO:tasks.workunit.client.0.smithi151.stdout:test api_snapshots on pid 91545 2023-11-02T08:37:26.338 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91545 2023-11-02T08:37:26.338 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.338 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.338 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_misc_pp: /' 2023-11-02T08:37:26.339 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-11-02T08:37:26.340 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_snapshots_pp 2023-11-02T08:37:26.340 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.340 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_snapshots_pp' 2023-11-02T08:37:26.340 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.341 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.341 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.341 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.341 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.341 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_snapshots_pp 2023-11-02T08:37:26.342 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.342 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_misc_pp 2023-11-02T08:37:26.342 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.342 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.342 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.343 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.343 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.343 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.344 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.344 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.344 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.344 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.344 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.345 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.345 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.345 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.346 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.346 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.346 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.346 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.346 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_snapshots_pp 2023-11-02T08:37:26.347 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91559 2023-11-02T08:37:26.347 INFO:tasks.workunit.client.0.smithi151.stdout:test api_snapshots_pp on pid 91559 2023-11-02T08:37:26.363 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_snapshots_pp on pid 91559' 2023-11-02T08:37:26.363 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91559 2023-11-02T08:37:26.363 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.363 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.364 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.364 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.364 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.364 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.364 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.364 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.365 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.365 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.365 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.365 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.366 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.366 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.366 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.366 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.366 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.367 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.367 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.367 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.367 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.367 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.368 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.368 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.368 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.368 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.369 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-11-02T08:37:26.369 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.369 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.369 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.369 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.370 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.370 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.370 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.370 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.370 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.371 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.371 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.371 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.371 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.371 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.372 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.372 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.372 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.372 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.373 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_stat 2023-11-02T08:37:26.373 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_stat' 2023-11-02T08:37:26.373 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.373 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.373 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.373 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.374 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.374 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.374 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.374 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.374 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.375 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.375 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.375 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.375 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.375 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.376 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.376 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.376 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.376 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.376 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.377 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.377 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.377 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.377 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.377 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.378 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.378 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.378 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.378 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.378 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.379 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.379 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.379 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.379 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.379 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.380 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.380 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.380 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.380 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.380 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.380 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.381 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.381 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.381 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.381 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.381 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.382 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.382 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.382 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.382 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.382 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.383 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.383 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.383 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.383 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.384 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.384 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_stat 2023-11-02T08:37:26.385 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.389 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.389 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.389 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.389 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.389 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.390 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.390 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.390 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.390 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.390 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.391 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.391 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.391 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.391 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.391 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.392 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.392 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.392 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.392 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.392 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.393 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.393 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_stat 2023-11-02T08:37:26.393 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91604 2023-11-02T08:37:26.393 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_stat on pid 91604' 2023-11-02T08:37:26.394 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.394 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.394 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.394 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.395 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.395 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.395 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.395 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.396 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.396 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.396 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.396 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.396 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.397 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.397 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.397 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.397 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.397 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.398 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.398 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.398 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.398 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.398 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.399 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.399 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.399 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.399 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.399 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.399 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.400 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.400 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.400 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.400 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.400 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.401 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.401 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.401 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.401 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.401 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.402 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.402 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.402 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.402 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.403 INFO:tasks.workunit.client.0.smithi151.stdout:test api_stat on pid 91604 2023-11-02T08:37:26.406 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91604 2023-11-02T08:37:26.407 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.407 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.407 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-11-02T08:37:26.407 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.407 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.408 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.408 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.408 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.408 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.408 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.409 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.409 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.409 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.409 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.409 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.410 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.410 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91447/exe 2023-11-02T08:37:26.410 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.410 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.410 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.411 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.411 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.411 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.411 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.411 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.411 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.412 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.412 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.412 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.412 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.413 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.413 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.413 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-11-02T08:37:26.413 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [----------] Global test environment set-up. 2023-11-02T08:37:26.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [----------] 12 tests from AsioRados 2023-11-02T08:37:26.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.004+0000 7fcbde4f9780 10 monclient: get_monmap_and_config 2023-11-02T08:37:26.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.004+0000 7fcbde4f9780 10 monclient: build_initial_monmap 2023-11-02T08:37:26.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.004+0000 7fcbde4f9780 10 monclient: monmap: 2023-11-02T08:37:26.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: epoch 0 2023-11-02T08:37:26.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:37:26.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: last_changed 0.000000 2023-11-02T08:37:26.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: created 0.000000 2023-11-02T08:37:26.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: min_mon_release 0 (unknown) 2023-11-02T08:37:26.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: election_strategy: 1 2023-11-02T08:37:26.416 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:37:26.416 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:37:26.416 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:37:26.416 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.004+0000 7fcbde4f9780 1 Processor -- start 2023-11-02T08:37:26.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.004+0000 7fcbde4f9780 1 -- start start 2023-11-02T08:37:26.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.004+0000 7fcbde4f9780 10 monclient: init 2023-11-02T08:37:26.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient: _reopen_session rank -1 2023-11-02T08:37:26.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:37:26.418 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 0x55776a467f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:37:26.423 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient(hunting): picked mon.noname-a con 0x55776a45f5e0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:37:26.424 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55776a45f9c0 0x55776a467150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:37:26.424 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient(hunting): picked mon.noname-c con 0x55776a45f9c0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:37:26.424 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 0x55776a4efc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:37:26.424 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient(hunting): picked mon.noname-b con 0x55776a4f08a0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:37:26.424 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient(hunting): start opening mon connection 2023-11-02T08:37:26.440 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776a379280 con 0x55776a45f5e0 2023-11-02T08:37:26.440 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient(hunting): start opening mon connection 2023-11-02T08:37:26.440 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776a378f20 con 0x55776a4f08a0 2023-11-02T08:37:26.441 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient(hunting): start opening mon connection 2023-11-02T08:37:26.441 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x55776a35c420 con 0x55776a45f9c0 2023-11-02T08:37:26.441 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient(hunting): _renew_subs 2023-11-02T08:37:26.444 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.005+0000 7fcbde4f9780 10 monclient(hunting): authenticate will time out at 1417.361274s 2023-11-02T08:37:26.448 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.449 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.450 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.450 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.450 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.451 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.451 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.452 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.453 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.454 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.454 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.454 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.454 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.454 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.455 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.455 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.455 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.455 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.455 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.456 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.456 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.456 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.456 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.456 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.457 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.457 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.457 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.457 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.457 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.457 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.458 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.458 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.458 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.458 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.458 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.459 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.460 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.461 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.461 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91500/exe 2023-11-02T08:37:26.462 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.462 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.462 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.463 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.463 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.464 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.464 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.464 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.464 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.465 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.006+0000 7fcbdc7f9640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 0x55776a4efc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:37:26.465 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.007+0000 7fcbdc7f9640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 0x55776a4efc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.151:50476/0 (socket says 172.21.15.151:50476) 2023-11-02T08:37:26.465 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.007+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/3060923965 learned_addr learned my addr 172.21.15.151:0/3060923965 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:37:26.465 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.007+0000 7fcbdc7f9640 10 monclient(hunting): get_auth_request con 0x55776a4f08a0 auth_method 0 2023-11-02T08:37:26.466 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.007+0000 7fcbdc7f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:37:26.466 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.007+0000 7fcbdc7f9640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:37:26.467 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.007+0000 7fcbdc7f9640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:37:26.467 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.008+0000 7fcbdc7f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:37:26.467 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.008+0000 7fcbdc7f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:37:26.467 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.008+0000 7fcbdc7f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:37:26.467 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.012+0000 7fcbdbff8640 1 --2- 172.21.15.151:0/3060923965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 0x55776a467f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:37:26.468 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.012+0000 7fcbdbff8640 10 monclient(hunting): get_auth_request con 0x55776a45f5e0 auth_method 0 2023-11-02T08:37:26.468 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.012+0000 7fcbdbff8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:37:26.468 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.012+0000 7fcbdbff8640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:37:26.468 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.012+0000 7fcbdbff8640 10 monclient(hunting): _init_auth creating new auth 2023-11-02T08:37:26.469 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.013+0000 7fcbdbff8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:37:26.469 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.013+0000 7fcbdbff8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:37:26.469 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.013+0000 7fcbdbff8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:37:26.469 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 10 monclient(hunting): handle_auth_done global_id 4435 payload 274 2023-11-02T08:37:26.469 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/3060923965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55776a45f9c0 msgr2=0x55776a467150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:37:26.470 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 1 --2- 172.21.15.151:0/3060923965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55776a45f9c0 0x55776a467150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:37:26.470 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/3060923965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 msgr2=0x55776a467f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-02T08:37:26.474 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.477 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.478 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.478 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_stat_pp 2023-11-02T08:37:26.479 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_stat_pp' 2023-11-02T08:37:26.479 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.479 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_tier_pp 2023-11-02T08:37:26.479 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-11-02T08:37:26.479 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_pool.log 2023-11-02T08:37:26.480 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_pool 2023-11-02T08:37:26.481 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_tier_pp: /' 2023-11-02T08:37:26.482 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.482 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.482 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.482 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.483 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.483 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.483 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.483 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.483 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.484 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.484 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.484 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.484 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.484 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.485 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.485 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.485 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.485 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_stat_pp 2023-11-02T08:37:26.485 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.485 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.486 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.486 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.486 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.486 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.487 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.487 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_stat_pp 2023-11-02T08:37:26.487 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91661 2023-11-02T08:37:26.487 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_stat_pp on pid 91661' 2023-11-02T08:37:26.489 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 1 --2- 172.21.15.151:0/3060923965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55test api_stat_pp on pid 91661 2023-11-02T08:37:26.495 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91661 2023-11-02T08:37:26.496 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.496 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.496 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.496 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.497 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.497 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.497 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.497 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.497 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.498 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.498 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.498 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.499 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.499 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.499 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.499 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.499 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_pool: /' 2023-11-02T08:37:26.500 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.500 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.500 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.500 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.500 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.501 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.501 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.501 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-11-02T08:37:26.501 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.501 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.502 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.502 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.502 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.503 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.503 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.503 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_watch_notify 2023-11-02T08:37:26.503 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.503 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.504 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.504 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.504 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.504 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.504 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.504 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.505 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.505 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.505 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.505 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.505 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.506 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.506 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.506 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.506 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.506 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.507 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.507 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.507 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.507 INFO:tasks.workunit.client.0.smithi151.stdout:776a45f5e0 0x55776a467f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:37:26.508 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 10 monclient: _finish_hunting 0 2023-11-02T08:37:26.508 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 1 monclient: found mon.noname-b 2023-11-02T08:37:26.508 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:37:26.508 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.178:3300/0 2023-11-02T08:37:26.508 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.014+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/3060923965 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc005b10 con 0x55776a4f08a0 2023-11-02T08:37:26.509 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.017+0000 7fcbdc7f9640 1 --2- 172.21.15.151:0/3060923965 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 0x55776a4efc40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc005e20 tx=0x7fcbcc053000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a92a0d866b167717 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:37:26.509 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 1 -- 172.21.15.151:0/3060923965 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0112f0 con 0x55776a4f08a0 2023-11-02T08:37:26.509 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:37:26.509 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-11-02T08:37:26.509 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 10 monclient: dump: 2023-11-02T08:37:26.510 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: epoch 1 2023-11-02T08:37:26.510 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:37:26.510 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:37:26.511 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:37:26.511 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: min_mon_release 18 (reef) 2023-11-02T08:37:26.511 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: election_strategy: 1 2023-11-02T08:37:26.511 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:37:26.511 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:37:26.512 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:37:26.512 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.512 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 10 monclient: _finish_auth 0 2023-11-02T08:37:26.512 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 10 monclient: _check_auth_tickets 2023-11-02T08:37:26.513 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:26.513 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 1 -- 172.21.15.151:0/3060923965 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc058940 con 0x55776a4f08a0 2023-11-02T08:37:26.513 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:37:26.513 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 1 -- 172.21.15.151:0/3060923965 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc058cf0 con 0x55776a4f08a0 2023-11-02T08:37:26.514 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:37:26.514 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-11-02T08:37:26.514 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbda7f5640 10 monclient: dump: 2023-11-02T08:37:26.514 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: epoch 1 2023-11-02T08:37:26.515 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:37:26.515 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:37:26.515 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:37:26.515 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: min_mon_release 18 (reef) 2023-11-02T08:37:26.515 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: election_strategy: 1 2023-11-02T08:37:26.515 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:37:26.522 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.522 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.522 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.522 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.523 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.523 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.523 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.524 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.524 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_watch_notify' 2023-11-02T08:37:26.524 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91545/exe 2023-11-02T08:37:26.524 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.524 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.525 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.525 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.525 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.525 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.525 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.526 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.526 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.526 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.526 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.526 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.527 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.527 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.527 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.527 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.528 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.528 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.528 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.528 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.528 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.529 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.529 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.529 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_watch_notify 2023-11-02T08:37:26.529 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.529 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.530 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.530 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.530 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.530 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.530 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.531 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.531 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.531 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.531 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.531 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.531 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.532 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.532 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.532 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.532 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.532 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.533 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.533 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.533 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.534 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.534 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.534 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.534 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.534 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.535 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.535 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.535 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.535 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.535 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.536 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.536 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.536 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.536 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.536 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.536 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.537 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.537 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.537 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.537 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.537 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.538 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.538 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.538 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.538 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-11-02T08:37:26.538 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_snapshots 2023-11-02T08:37:26.539 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:37:26.539 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:37:26.539 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.539 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbde4f9780 5 monclient: authenticate success, global_id 4435 2023-11-02T08:37:26.540 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbde4f9780 10 monclient: get_monmap_and_config success 2023-11-02T08:37:26.540 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.026+0000 7fcbde4f9780 4 set_mon_vals no callback set 2023-11-02T08:37:26.540 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.027+0000 7fcbde4f9780 10 monclient: shutdown 2023-11-02T08:37:26.540 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.027+0000 7fcbde4f9780 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:37:26.540 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.027+0000 7fcbde4f9780 1 -- 172.21.15.151:0/3060923965 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 msgr2=0x55776a4efc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:37:26.541 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.027+0000 7fcbde4f9780 1 --2- 172.21.15.151:0/3060923965 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 0x55776a4efc40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc005e20 tx=0x7fcbcc053000 comp rx=0 tx=0).stop 2023-11-02T08:37:26.541 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.031+0000 7fcbde4f9780 1 -- 172.21.15.151:0/3060923965 shutdown_connections 2023-11-02T08:37:26.541 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.031+0000 7fcbde4f9780 1 --2- 172.21.15.151:0/3060923965 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 0x55776a4efc40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:37:26.541 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.031+0000 7fcbde4f9780 1 --2- 172.21.15.151:0/3060923965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55776a45f9c0 0x55776a467150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:37:26.542 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.031+0000 7fcbde4f9780 1 --2- 172.21.15.151:0/3060923965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 0x55776a467f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:37:26.542 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.031+0000 7fcbde4f9780 1 -- 172.21.15.151:0/3060923965 >> 172.21.15.151:0/3060923965 conn(0x55776a424440 msgr2=0x55776a4600d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:37:26.542 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.033+0000 7fcbde4f9780 1 -- 172.21.15.151:0/3060923965 shutdown_connections 2023-11-02T08:37:26.542 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.033+0000 7fcbde4f9780 1 -- 172.21.15.151:0/3060923965 wait complete. 2023-11-02T08:37:26.542 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.033+0000 7fcbde4f9780 10 monclient: build_initial_monmap 2023-11-02T08:37:26.542 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.033+0000 7fcbde4f9780 10 monclient: monmap: 2023-11-02T08:37:26.543 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: epoch 0 2023-11-02T08:37:26.543 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-11-02T08:37:26.543 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: last_changed 0.000000 2023-11-02T08:37:26.543 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: created 0.000000 2023-11-02T08:37:26.543 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: min_mon_release 0 (unknown) 2023-11-02T08:37:26.544 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: election_strategy: 1 2023-11-02T08:37:26.544 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.noname-a 2023-11-02T08:37:26.544 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.noname-b 2023-11-02T08:37:26.545 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-c 2023-11-02T08:37:26.545 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.545 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.034+0000 7fcbde4f9780 1 librados: starting msgr at 2023-11-02T08:37:26.545 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.034+0000 7fcbde4f9780 1 librados: starting objecter 2023-11-02T08:37:26.545 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.034+0000 7fcbde4f9780 1 Processor -- start 2023-11-02T08:37:26.546 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.034+0000 7fcbde4f9780 1 -- start start 2023-11-02T08:37:26.546 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.034+0000 7fcbde4f9780 1 librados: setting wanted keys 2023-11-02T08:37:26.546 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.034+0000 7fcbde4f9780 1 librados: calling monclient init 2023-11-02T08:37:26.548 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_snapshots: /' 2023-11-02T08:37:26.548 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.548 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.548 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.548 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.549 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.549 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.549 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.549 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.549 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.550 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_watch_notify 2023-11-02T08:37:26.550 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91705 2023-11-02T08:37:26.550 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_watch_notify on pid 91705' 2023-11-02T08:37:26.550 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-11-02T08:37:26.551 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08test api_watch_notify on pid 91705 2023-11-02T08:37:26.554 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91705 2023-11-02T08:37:26.555 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.555 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.555 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_watch_notify_pp 2023-11-02T08:37:26.555 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_watch_notify_pp' 2023-11-02T08:37:26.555 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.556 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.556 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.556 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.556 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.556 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.557 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.557 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_watch_notify_pp 2023-11-02T08:37:26.557 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.557 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.557 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.557 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.558 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.558 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.558 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.559 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.559 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.559 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.559 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.559 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.559 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.560 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.560 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.560 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.560 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.560 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.561 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_watch_notify_pp 2023-11-02T08:37:26.561 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91716 2023-11-02T08:37:26.561 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_watch_notify_pp on pid 91716' 2023-11-02T08:37:26.562 INFO:tasks.workunit.client.0.smithi151.stdout:test api_watch_notify_pp on pid 91716 2023-11-02T08:37:26.569 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91716 2023-11-02T08:37:26.569 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.569 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.569 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_cmd 2023-11-02T08:37:26.570 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.570 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.570 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.570 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.570 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.571 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.571 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.571 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.571 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.571 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-11-02T08:37:26.572 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.572 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.572 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.572 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.572 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.573 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_cmd' 2023-11-02T08:37:26.573 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_cmd 2023-11-02T08:37:26.573 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.573 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.573 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.573 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.574 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.574 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.574 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.575 INFO:tasks.workunit.client.0.smithi151.stdout::37:26.034+0000 7fcbde4f9780 10 monclient: init 2023-11-02T08:37:26.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient: _reopen_session rank -1 2023-11-02T08:37:26.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient: _add_conns ranks=[0,2,1] 2023-11-02T08:37:26.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 0x55776a58c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:37:26.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient(hunting): picked mon.noname-a con 0x55776a45f5e0 addr [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] 2023-11-02T08:37:26.576 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55776a45f9c0 0x55776a4efa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:37:26.576 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient(hunting): picked mon.noname-c con 0x55776a45f9c0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T08:37:26.576 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 0x55776a4e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:37:26.576 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient(hunting): picked mon.noname-b con 0x55776a4f08a0 addr [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-11-02T08:37:26.576 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient(hunting): start opening mon connection 2023-11-02T08:37:26.577 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776a4e3580 con 0x55776a45f5e0 2023-11-02T08:37:26.577 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient(hunting): start opening mon connection 2023-11-02T08:37:26.577 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776a58d1c0 con 0x55776a4f08a0 2023-11-02T08:37:26.578 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient(hunting): start opening mon connection 2023-11-02T08:37:26.578 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x55776a427950 con 0x55776a45f9c0 2023-11-02T08:37:26.578 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient(hunting): _renew_subs 2023-11-02T08:37:26.578 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.035+0000 7fcbde4f9780 10 monclient(hunting): authenticate will time out at 1417.391192s 2023-11-02T08:37:26.578 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.047+0000 7fcbdbff8640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 0x55776a58c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:37:26.579 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.048+0000 7fcbdbff8640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 0x55776a58c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:34126/0 (socket says 172.21.15.151:34126) 2023-11-02T08:37:26.579 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.048+0000 7fcbdbff8640 1 -- 172.21.15.151:0/4083040167 learned_addr learned my addr 172.21.15.151:0/4083040167 (peer_addr_for_me v2:172.21.15.151:0/0) 2023-11-02T08:37:26.579 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.048+0000 7fcbdbff8640 10 monclient(hunting): get_auth_request con 0x55776a45f5e0 auth_method 0 2023-11-02T08:37:26.579 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.048+0000 7fcbdbff8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-11-02T08:37:26.579 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.048+0000 7fcbdbff8640 10 monclient(hunting): _init_auth method 2 2023-11-02T08:37:26.587 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.587 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.588 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.588 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.588 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.588 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.588 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.589 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.589 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.589 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.589 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.589 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.590 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.590 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.590 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.590 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.590 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.591 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.591 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.591 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.591 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.592 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.592 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91559/exe 2023-11-02T08:37:26.592 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.592 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.592 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.593 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.593 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.593 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.593 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.593 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.593 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.594 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.594 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.594 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.594 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.594 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.595 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.595 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.595 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.595 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.596 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.596 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.596 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.596 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.596 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.597 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.597 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.597 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.597 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.597 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.598 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.598 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.598 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.599 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.599 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.599 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.599 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.599 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.600 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.600 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.600 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91604/exe 2023-11-02T08:37:26.600 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.600 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.601 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.601 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.601 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.601 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.601 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.602 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.602 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.602 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.602 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.602 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.603 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.603 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.603 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.603 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.603 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.603 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.604 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.604 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.604 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.604 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.604 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_snapshots_pp 2023-11-02T08:37:26.605 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.605 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.605 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_cmd 2023-11-02T08:37:26.605 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91743 2023-11-02T08:37:26.605 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_cmd on pid 91743' 2023-11-02T08:37:26.605 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-11-02T08:37:26.606 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.048+0000 7fcbdbff8640 10 monclient(hunting):test api_cmd on pid 91743 2023-11-02T08:37:26.619 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91743 2023-11-02T08:37:26.620 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.620 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.620 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.620 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.620 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.621 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.621 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.621 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.621 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.621 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.622 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_cmd_pp 2023-11-02T08:37:26.622 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.622 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.622 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.622 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.623 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_cmd_pp' 2023-11-02T08:37:26.623 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.623 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.623 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.623 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.624 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.624 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.624 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.624 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.624 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.625 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.625 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.625 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.625 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.625 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.625 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.626 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.626 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.626 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.626 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.626 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.627 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.627 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.627 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.627 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.627 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.628 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.628 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.628 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.628 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.628 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.628 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.629 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.629 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.629 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.629 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.629 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.630 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.630 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-11-02T08:37:26.630 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-11-02T08:37:26.630 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.630 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.631 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.631 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.631 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.631 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.631 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.632 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.632 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.632 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.632 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.632 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.632 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.633 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.633 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.633 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.633 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.633 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.634 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_stat 2023-11-02T08:37:26.634 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_stat.log 2023-11-02T08:37:26.634 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_stat: /' 2023-11-02T08:37:26.634 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.634 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.635 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.635 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.635 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.635 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.635 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.635 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.636 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.636 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.636 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.636 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.636 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.637 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.637 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.637 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.637 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.637 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.638 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.638 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.638 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.638 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.638 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.638 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.639 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.639 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.639 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.639 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.639 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.640 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.640 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.640 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.640 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.640 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.640 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.641 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.641 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.641 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.641 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.641 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.642 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.642 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.642 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91661/exe 2023-11-02T08:37:26.642 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.642 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.643 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.643 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.643 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.643 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.643 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.643 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.644 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.644 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.644 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.644 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.644 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.645 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.645 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.645 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.645 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.645 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.646 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.646 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.646 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.646 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.646 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.647 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.647 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.647 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.647 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.647 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.647 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.648 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.648 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.649 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.649 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.649 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.649 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.649 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_cmd_pp 2023-11-02T08:37:26.650 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.650 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.650 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.650 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.650 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.651 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.651 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.651 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.651 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.651 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.652 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.652 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.652 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.652 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.652 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.652 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.653 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.653 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.653 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.653 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.653 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.654 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.654 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.654 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.654 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.654 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.655 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.655 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.655 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.655 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.655 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.656 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.656 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_cmd_pp 2023-11-02T08:37:26.656 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91790 2023-11-02T08:37:26.656 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_cmd_pp on pid 91790' 2023-11-02T08:37:26.657 INFO:tasks.workunit.client.0.smithi151.stdout:test api_cmd_pp on pid 91790 2023-11-02T08:37:26.676 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91790 2023-11-02T08:37:26.677 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.677 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.677 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_stat_pp 2023-11-02T08:37:26.677 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-11-02T08:37:26.677 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.678 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.678 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-11-02T08:37:26.678 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.678 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.678 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.679 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_stat_pp: /' 2023-11-02T08:37:26.679 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.679 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.679 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.679 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.680 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.680 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.680 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.680 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.680 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.680 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.681 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.681 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.681 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.681 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.682 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.682 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.682 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.682 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.682 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.683 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.683 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.683 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.683 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.683 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.683 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.684 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.684 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.684 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.684 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.684 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.685 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.685 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.685 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.685 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.685 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.686 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.686 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.686 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.686 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.686 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.687 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.687 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.687 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.687 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.687 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.687 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.688 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.688 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.688 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.688 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.688 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.689 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.689 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.689 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.690 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.690 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.690 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.690 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.690 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.691 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.691 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.691 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.691 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.691 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.691 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.692 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.692 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.692 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.693 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.693 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.693 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.693 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.693 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.693 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_service 2023-11-02T08:37:26.694 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.694 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91716/exe 2023-11-02T08:37:26.694 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_service' 2023-11-02T08:37:26.694 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.694 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.695 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.695 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.695 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.695 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.695 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.696 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.696 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.696 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.696 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.696 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.697 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.697 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.697 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.697 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.697 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_watch_notify_pp 2023-11-02T08:37:26.697 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.698 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.698 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.698 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.698 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.698 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.699 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.699 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.699 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.699 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.699 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.700 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.700 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-11-02T08:37:26.700 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.700 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.700 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.700 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.701 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.701 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.701 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.701 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.701 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.702 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.702 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.702 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.702 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.702 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.703 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.703 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.703 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.703 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.703 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.703 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.704 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.704 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.704 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.704 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.704 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.705 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.705 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.705 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.705 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.705 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.706 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.706 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.706 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.706 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.706 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.707 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.707 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.707 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.707 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.707 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.707 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.708 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.708 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.708 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.708 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.708 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.709 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.709 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.709 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.709 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.709 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.710 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.710 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.710 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.710 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.711 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.711 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.711 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.711 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.712 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.712 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.712 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.712 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91743/exe 2023-11-02T08:37:26.712 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.713 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.713 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.713 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.713 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.713 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.713 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.714 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.714 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.714 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.714 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.714 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.715 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.715 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.715 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.715 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.715 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.716 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.716 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.716 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.716 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.716 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.716 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.717 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.717 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.717 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.717 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.717 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_cmd 2023-11-02T08:37:26.718 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.718 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_cmd: /' 2023-11-02T08:37:26.718 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.718 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-11-02T08:37:26.718 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.719 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.719 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.719 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.719 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.719 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.719 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.720 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.720 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.720 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.720 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.720 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.721 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.721 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.721 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.721 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.721 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.722 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.722 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.722 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.722 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.722 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.722 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91705/exe 2023-11-02T08:37:26.723 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.723 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.723 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.723 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_service 2023-11-02T08:37:26.723 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.724 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.724 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.724 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.724 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.724 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.725 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_service 2023-11-02T08:37:26.725 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.725 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.725 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91842 2023-11-02T08:37:26.725 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_service on pid 91842' 2023-11-02T08:37:26.726 INFO:tasks.workunit.client.0.smithi151.stdout:test api_service on pid 91842 2023-11-02T08:37:26.736 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91842 2023-11-02T08:37:26.736 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.736 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.736 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-11-02T08:37:26.737 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_watch_notify 2023-11-02T08:37:26.737 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.737 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.737 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.737 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_cmd.log 2023-11-02T08:37:26.737 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.738 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.738 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.738 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-11-02T08:37:26.738 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.738 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.739 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.739 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.739 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.739 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.739 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.740 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.740 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.740 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.740 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.740 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.741 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.741 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.741 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.741 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.741 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.742 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.742 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.742 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.742 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.742 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.742 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.743 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.743 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.743 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.743 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.743 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.744 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91790/exe 2023-11-02T08:37:26.744 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.744 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.744 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.744 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.745 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.745 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.745 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.745 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.745 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.745 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.746 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.746 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.746 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.746 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.746 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.747 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.747 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.747 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.747 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.747 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.748 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.748 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.748 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.748 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.748 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.748 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.749 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.749 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.749 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.749 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.749 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_watch_notify: /' 2023-11-02T08:37:26.750 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_cmd_pp 2023-11-02T08:37:26.750 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.750 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.750 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.750 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.751 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.751 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.751 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.751 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.751 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.751 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.752 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.752 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.752 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.752 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.752 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.753 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.753 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.753 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.753 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.753 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.754 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.754 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.754 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.754 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.754 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.754 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.755 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.755 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-11-02T08:37:26.755 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.755 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.755 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-11-02T08:37:26.756 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_service_pp 2023-11-02T08:37:26.756 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_service_pp' 2023-11-02T08:37:26.756 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.756 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.756 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.757 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.757 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.757 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.757 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.757 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.757 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.758 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.758 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.758 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.758 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.758 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.759 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.759 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.759 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.759 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_service_pp 2023-11-02T08:37:26.760 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.760 INFO:tasks.workunit.client.0.smithi151.stdout: _init_auth creating new auth 2023-11-02T08:37:26.760 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.048+0000 7fcbdbff8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-11-02T08:37:26.760 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.048+0000 7fcbdbff8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-11-02T08:37:26.761 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.048+0000 7fcbdbff8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-11-02T08:37:26.761 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2023-11-02T08:37:26.761 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 1 -- 172.21.15.151:0/4083040167 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55776a45f9c0 msgr2=0x55776a4efa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:37:26.762 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55776a45f9c0 0x55776a4efa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:37:26.762 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 1 -- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 msgr2=0x55776a4e5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:37:26.762 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55776a4f08a0 0x55776a4e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:37:26.762 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 10 monclient: _finish_hunting 0 2023-11-02T08:37:26.763 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 1 monclient: found mon.noname-a 2023-11-02T08:37:26.763 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:37:26.763 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.151:3300/0 2023-11-02T08:37:26.763 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.053+0000 7fcbdbff8640 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55776a3a24e0 con 0x55776a45f5e0 2023-11-02T08:37:26.763 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.056+0000 7fcbdbff8640 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 0x55776a58c270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400d500 tx=0x7fcbd400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:37:26.764 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.056+0000 7fcbd97f3640 10 client.?.objecter ms_handle_connect 0x55776a45f5e0 2023-11-02T08:37:26.764 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.056+0000 7fcbd97f3640 10 client.?.objecter resend_mon_ops 2023-11-02T08:37:26.764 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== mon.0 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4011600 con 0x55776a45f5e0 2023-11-02T08:37:26.764 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:37:26.764 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-11-02T08:37:26.765 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 10 monclient: dump: 2023-11-02T08:37:26.765 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: epoch 1 2023-11-02T08:37:26.765 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:37:26.765 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:37:26.765 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:37:26.765 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: min_mon_release 18 (reef) 2023-11-02T08:37:26.766 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: election_strategy: 1 2023-11-02T08:37:26.766 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:37:26.766 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:37:26.770 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.770 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.771 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.771 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.771 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.771 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.771 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_service_pp 2023-11-02T08:37:26.772 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.772 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91901 2023-11-02T08:37:26.772 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:679test api_service_pp on pid 91901 2023-11-02T08:37:26.778 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_service_pp on pid 91901' 2023-11-02T08:37:26.778 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91901 2023-11-02T08:37:26.778 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.778 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.779 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-11-02T08:37:26.779 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.779 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.779 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.779 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.780 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.780 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.780 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.780 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.780 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.781 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.781 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.781 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.781 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.781 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.781 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.782 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.782 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.782 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.782 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.782 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.783 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.783 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.783 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.783 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.783 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.784 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.784 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.784 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.784 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.784 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.785 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.785 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.785 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.785 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.785 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.785 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.786 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.786 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.786 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.786 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.786 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.787 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.787 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.787 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.787 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.788 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.788 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_c_write_operations 2023-11-02T08:37:26.788 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_c_write_operations' 2023-11-02T08:37:26.788 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91842/exe 2023-11-02T08:37:26.788 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.789 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_c_write_operations 2023-11-02T08:37:26.789 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.789 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.789 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.790 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.790 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.790 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.790 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.790 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.791 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.791 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.791 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.791 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_service 2023-11-02T08:37:26.791 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.792 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_c_write_operations 2023-11-02T08:37:26.792 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.792 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.792 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91921 2023-11-02T08:37:26.793 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_c_write_operations on pid 91921' 2023-11-02T08:37:26.793 INFO:tasks.workunit.client.0.smithi151.stdout:test api_c_write_operations on pid 91921 2023-11-02T08:37:26.798 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91921 2023-11-02T08:37:26.799 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.799 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.799 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.799 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.799 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.800 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.800 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.800 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.800 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.800 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.801 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.801 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.801 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.801 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.801 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.801 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.802 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.802 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.802 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.802 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_c_read_operations 2023-11-02T08:37:26.802 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_c_read_operations' 2023-11-02T08:37:26.803 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-11-02T08:37:26.803 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.803 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_service.log 2023-11-02T08:37:26.803 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.803 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.804 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.804 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.804 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.804 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_service: /' 2023-11-02T08:37:26.804 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_c_read_operations 2023-11-02T08:37:26.805 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.805 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.805 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.805 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.805 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.805 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.806 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.806 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.806 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.806 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.806 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.807 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.807 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.807 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.807 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.807 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.808 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.808 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.808 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.808 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.808 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.808 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.809 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.809 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.809 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.809 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.809 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.810 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.810 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.810 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_c_read_operations 2023-11-02T08:37:26.810 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91940 2023-11-02T08:37:26.811 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_c_read_operations on pid 91940' 2023-11-02T08:37:26.811 INFO:tasks.workunit.client.0.smithi151.stdout:test api_c_read_operations on pid 91940 2023-11-02T08:37:26.817 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91940 2023-11-02T08:37:26.818 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.818 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.818 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.818 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.818 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s api_cls_remote_reads 2023-11-02T08:37:26.819 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.819 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.819 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' api_cls_remote_reads' 2023-11-02T08:37:26.819 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.819 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.820 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.820 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.820 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.820 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.820 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.821 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.821 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-11-02T08:37:26.821 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.821 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.821 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.822 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.822 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.822 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.822 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.822 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.822 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.823 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.823 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.823 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.823 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.823 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.824 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.824 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.824 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.824 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.824 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.825 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.825 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.825 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.825 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.825 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.825 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.826 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.826 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.826 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.826 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.826 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.827 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.827 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.827 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.827 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.827 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.828 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.828 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.828 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.828 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.828 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.828 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.829 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.829 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.829 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.829 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.829 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.830 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.830 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.830 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.830 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.830 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.831 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.831 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.831 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.831 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.831 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.831 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.832 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.832 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo api_cls_remote_reads 2023-11-02T08:37:26.832 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=api_cls_remote_reads 2023-11-02T08:37:26.832 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91960 2023-11-02T08:37:26.832 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test api_cls_remote_reads on pid 91960' 2023-11-02T08:37:26.833 INFO:tasks.workunit.client.0.smithi151.stdout:test api_cls_remote_reads on pid 91960 2023-11-02T08:37:26.840 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91960 2023-11-02T08:37:26.840 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.841 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.841 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.841 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.841 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.841 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s list_parallel 2023-11-02T08:37:26.842 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.842 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.842 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.842 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.842 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.842 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.843 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.843 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.843 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.843 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.843 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.844 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.844 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.844 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.844 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.844 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.845 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.845 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.845 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.845 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.845 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.845 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.846 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.846 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.846 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.846 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.846 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.847 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.847 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.847 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.847 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.847 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.848 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.848 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.848 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.848 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.848 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.849 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.849 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.849 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.849 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-11-02T08:37:26.849 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.849 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.850 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.850 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.850 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.850 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.850 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.851 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' list_parallel' 2023-11-02T08:37:26.851 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91901/exe 2023-11-02T08:37:26.851 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.851 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo list_parallel 2023-11-02T08:37:26.851 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.851 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.852 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.852 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.852 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.852 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.852 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.853 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.853 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.853 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.853 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.853 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.854 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.854 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.854 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.854 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.854 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.855 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.855 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.855 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.855 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.855 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.855 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.856 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.856 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.856 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.856 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.856 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.857 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.857 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.857 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.857 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.857 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.858 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.858 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.858 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.858 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=list_parallel 2023-11-02T08:37:26.858 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91985 2023-11-02T08:37:26.859 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test list_parallel on pid 91985' 2023-11-02T08:37:26.859 INFO:tasks.workunit.client.0.smithi151.stdout:test list_parallel on pid 91985 2023-11-02T08:37:26.867 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=91985 2023-11-02T08:37:26.867 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_service_pp 2023-11-02T08:37:26.867 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.868 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-11-02T08:37:26.868 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.868 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s open_pools_parallel 2023-11-02T08:37:26.868 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.868 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.869 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' open_pools_parallel' 2023-11-02T08:37:26.869 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.869 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.869 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.869 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.870 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.870 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.870 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.870 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.870 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.871 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.871 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.871 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.871 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_service_pp: /' 2023-11-02T08:37:26.871 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.871 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.872 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.872 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.872 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.872 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.872 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.873 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.873 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.873 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.873 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-11-02T08:37:26.873 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.874 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.874 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.874 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.874 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.874 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.875 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.875 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.875 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.875 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo open_pools_parallel 2023-11-02T08:37:26.875 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.875 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.876 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.876 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.876 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.876 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.876 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.877 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.877 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.877 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.877 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.877 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.877 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.878 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.878 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.878 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.878 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.878 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.879 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.879 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.879 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.879 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.879 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.880 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.880 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.880 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.880 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.880 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.880 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.881 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.881 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.881 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.881 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.881 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.882 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.882 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.882 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.882 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.882 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.883 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.883 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.883 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.883 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.883 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.884 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.884 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.884 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.884 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.884 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.884 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.885 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.885 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.885 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.885 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.885 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.886 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.886 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.886 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.886 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.886 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.886 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.887 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.887 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.887 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=open_pools_parallel 2023-11-02T08:37:26.887 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=92006 2023-11-02T08:37:26.887 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test open_pools_parallel on pid 92006' 2023-11-02T08:37:26.888 INFO:tasks.workunit.client.0.smithi151.stdout:test open_pools_parallel on pid 92006 2023-11-02T08:37:26.898 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=92006 2023-11-02T08:37:26.898 INFO:tasks.workunit.client.0.smithi151.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-11-02T08:37:26.899 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.899 INFO:tasks.workunit.client.0.smithi151.stderr:++ printf %25s delete_pools_parallel 2023-11-02T08:37:26.899 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.899 INFO:tasks.workunit.client.0.smithi151.stderr:+ r=' delete_pools_parallel' 2023-11-02T08:37:26.899 INFO:tasks.workunit.client.0.smithi151.stderr:++ echo delete_pools_parallel 2023-11-02T08:37:26.900 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.900 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.900 INFO:tasks.workunit.client.0.smithi151.stderr:++ awk '{print $1}' 2023-11-02T08:37:26.900 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.900 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.901 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.901 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.901 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.901 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.901 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.902 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.902 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.902 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.902 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.902 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.902 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.903 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.903 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.903 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.903 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.903 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.904 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.904 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.904 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.904 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.904 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.905 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.905 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.905 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91921/exe 2023-11-02T08:37:26.905 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.905 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.905 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.906 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.906 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.906 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-11-02T08:37:26.906 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.906 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.907 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.907 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.907 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.907 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.907 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.908 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.908 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.908 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.908 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.908 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.909 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.909 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.909 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.909 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.909 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.909 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.910 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.910 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.910 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.910 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.910 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.911 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.911 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.911 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.912 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.912 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.912 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.912 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.912 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.913 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.913 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.913 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.913 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.913 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.914 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.914 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.914 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.914 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.914 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.915 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.915 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.915 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.915 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.915 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.915 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.916 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.916 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.916 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_c_write_operations 2023-11-02T08:37:26.916 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.916 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-11-02T08:37:26.917 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.917 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.917 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.917 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.917 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.918 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.918 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.918 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.918 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91940/exe 2023-11-02T08:37:26.918 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.919 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.919 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.919 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.919 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.919 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-11-02T08:37:26.919 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.920 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.920 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.920 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.920 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.920 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.921 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.921 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.921 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.921 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.921 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.922 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.922 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.922 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.922 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.922 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.922 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.923 INFO:tasks.workunit.client.0.smithi151.stderr:+ ff=delete_pools_parallel 2023-11-02T08:37:26.923 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.923 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.923 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=92035 2023-11-02T08:37:26.923 INFO:tasks.workunit.client.0.smithi151.stderr:+ echo 'test delete_pools_parallel on pid 92035' 2023-11-02T08:37:26.924 INFO:tasks.workunit.client.0.smithi151.stdout:test delete_pools_parallel on pid 92035 2023-11-02T08:37:26.950 INFO:tasks.workunit.client.0.smithi151.stderr:+ pids[$f]=92035 2023-11-02T08:37:26.951 INFO:tasks.workunit.client.0.smithi151.stderr:+ ret=0 2023-11-02T08:37:26.951 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' 1 -eq 1 ']' 2023-11-02T08:37:26.951 INFO:tasks.workunit.client.0.smithi151.stderr:+ for t in "${!pids[@]}" 2023-11-02T08:37:26.952 INFO:tasks.workunit.client.0.smithi151.stderr:+ pid=91447 2023-11-02T08:37:26.952 INFO:tasks.workunit.client.0.smithi151.stderr:+ wait 91447 2023-11-02T08:37:26.952 INFO:tasks.workunit.client.0.smithi151.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-11-02T08:37:26.952 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.953 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.953 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.953 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.953 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.953 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.954 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.954 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -f /etc/bashrc ']' 2023-11-02T08:37:26.954 INFO:tasks.workunit.client.0.smithi151.stderr:+ . /etc/bashrc 2023-11-02T08:37:26.954 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.955 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.955 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.955 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -z '' ']' 2023-11-02T08:37:26.955 INFO:tasks.workunit.client.0.smithi151.stderr:++ BASHRCSOURCED=Y 2023-11-02T08:37:26.955 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.955 INFO:tasks.workunit.client.0.smithi151.stderr:++ shopt -q login_shell 2023-11-02T08:37:26.956 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_c_read_operations 2023-11-02T08:37:26.956 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.956 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-11-02T08:37:26.956 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.956 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.957 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.957 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.957 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.957 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.957 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-11-02T08:37:26.958 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.958 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.958 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.958 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.959 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.959 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.959 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.960 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.960 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.960 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.960 INFO:tasks.workunit.client.0.smithi151.stderr:+++ umask 2023-11-02T08:37:26.961 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.961 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.961 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.961 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.961 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.962 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.962 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.962 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.962 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.963 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.963 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.963 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.963 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.964 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.964 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.964 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.964 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.965 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.965 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.965 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.966 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.966 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.966 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.967 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.967 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.967 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.968 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.968 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.968 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.968 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' 0022 -eq 0 ']' 2023-11-02T08:37:26.969 INFO:tasks.workunit.client.0.smithi151.stderr:++ SHELL=/bin/bash 2023-11-02T08:37:26.970 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.970 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-11-02T08:37:26.970 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.970 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/bash_completion.sh 2023-11-02T08:37:26.970 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-11-02T08:37:26.971 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.971 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-11-02T08:37:26.971 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.971 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/cobbler.sh 2023-11-02T08:37:26.971 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.972 INFO:tasks.workunit.client.0.smithi151.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-11-02T08:37:26.972 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.972 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-11-02T08:37:26.972 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.973 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorgrep.sh 2023-11-02T08:37:26.973 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.973 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.974 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.974 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.974 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.974 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.974 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.975 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.978 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.978 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.978 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.979 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.980 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:26.980 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.980 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:26.980 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.981 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:26.981 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:26.981 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.981 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.981 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.981 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.982 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.982 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.982 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:26.982 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'grep=grep --color=auto' 2023-11-02T08:37:26.982 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'egrep=egrep --color=auto' 2023-11-02T08:37:26.983 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-11-02T08:37:26.983 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.983 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:26.983 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:26.983 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:26.984 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.984 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:26.984 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.984 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:26.984 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:26.985 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.985 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-11-02T08:37:26.985 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.985 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:26.986 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:26.986 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.987 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:26.987 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorls.sh 2023-11-02T08:37:26.987 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.987 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:26.988 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' '!' -t 0 ']' 2023-11-02T08:37:26.988 INFO:tasks.workunit.client.0.smithi151.stderr:+++ return 2023-11-02T08:37:26.988 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.988 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-11-02T08:37:26.988 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.989 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-11-02T08:37:26.989 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export S_COLORS=auto 2023-11-02T08:37:26.989 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:26.989 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.989 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:26.989 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.990 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:26.990 INFO:tasks.workunit.client.0.smithi151.stderr:+++ S_COLORS=auto 2023-11-02T08:37:26.991 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:26.991 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.991 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-11-02T08:37:26.991 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.991 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-11-02T08:37:26.991 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.992 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91985/exe 2023-11-02T08:37:26.992 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:26.992 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:26.992 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:26.992 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:26.993 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:26.993 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:26.993 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:26.993 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:26.993 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:26.994 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:26.994 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:26.994 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.994 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:26.994 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:26.995 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.995 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:26.995 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:26.995 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:26.995 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:26.995 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:26.996 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:26.996 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_list_parallel.log 2023-11-02T08:37:26.996 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ list_parallel: /' 2023-11-02T08:37:26.996 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_list_parallel 2023-11-02T08:37:26.996 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:26.997 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:26.997 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:26.997 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:26.997 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:26.997 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:26.997 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:26.998 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:26.998 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.998 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:26.998 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.998 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:26.999 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:26.999 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:26.999 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:26.999 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:26.999 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:27.000 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.000 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:27.000 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.001 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:27.001 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:27.001 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:27.001 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:27.001 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:27.002 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:27.002 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:27.002 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:27.006 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:27.006 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-11-02T08:37:27.006 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-11-02T08:37:27.006 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-11-02T08:37:27.007 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.007 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-11-02T08:37:27.007 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.007 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-11-02T08:37:27.007 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-11-02T08:37:27.008 INFO:tasks.workunit.client.0.smithi151.stderr:+++ /usr/libexec/grepconf.sh -c 2023-11-02T08:37:27.008 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:27.008 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:27.008 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:27.009 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:27.009 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:27.009 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:27.009 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:27.009 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:27.009 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:27.010 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:27.010 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:27.010 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:27.010 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.010 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:27.011 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:27.011 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:27.011 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:27.011 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:27.011 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.012 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:27.012 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:27.012 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:27.012 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.013 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:27.013 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.013 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:27.013 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:27.014 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:27.014 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:27.014 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:27.014 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:27.014 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:27.015 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:27.015 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.015 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:27.016 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.016 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:27.016 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:27.017 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.017 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:27.017 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.018 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:27.018 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:27.018 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/92035/exe 2023-11-02T08:37:27.019 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/91960/exe 2023-11-02T08:37:27.019 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:27.019 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:27.019 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:27.020 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:27.020 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:27.020 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:27.020 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:27.020 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:27.021 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:27.021 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:27.021 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:27.021 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:27.021 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-11-02T08:37:27.022 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-11-02T08:37:27.022 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:27.022 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:27.022 INFO:tasks.workunit.client.0.smithi151.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-11-02T08:37:27.022 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:27.023 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.024 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-11-02T08:37:27.025 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.025 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/debuginfod.sh 2023-11-02T08:37:27.025 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:27.025 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-11-02T08:37:27.026 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.026 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-11-02T08:37:27.026 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.026 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/gawk.sh 2023-11-02T08:37:27.027 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:27.027 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:27.027 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.027 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-11-02T08:37:27.027 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.028 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/lang.sh 2023-11-02T08:37:27.028 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup 2023-11-02T08:37:27.028 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:27.028 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_delete_pools_parallel 2023-11-02T08:37:27.028 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:27.029 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:27.029 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:27.031 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-11-02T08:37:27.031 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-11-02T08:37:27.031 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/locale 2023-11-02T08:37:27.032 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-11-02T08:37:27.032 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-11-02T08:37:27.032 INFO:tasks.workunit.client.0.smithi151.stdout:0/0] mon.c 2023-11-02T08:37:27.032 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.033 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 10 monclient: _finish_auth 0 2023-11-02T08:37:27.033 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 10 monclient: _check_auth_tickets 2023-11-02T08:37:27.033 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:27.033 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== mon.0 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd4011760 con 0x55776a45f5e0 2023-11-02T08:37:27.033 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 10 monclient: handle_config config(0 keys) v1 2023-11-02T08:37:27.034 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== mon.0 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4011b10 con 0x55776a45f5e0 2023-11-02T08:37:27.034 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-11-02T08:37:27.034 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-11-02T08:37:27.034 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd97f3640 10 monclient: dump: 2023-11-02T08:37:27.034 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: epoch 1 2023-11-02T08:37:27.035 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:37:27.035 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: last_changed 2023-11-02T08:34:48.026360+0000 2023-11-02T08:37:27.035 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: created 2023-11-02T08:34:48.026360+0000 2023-11-02T08:37:27.035 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: min_mon_release 18 (reef) 2023-11-02T08:37:27.035 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: election_strategy: 1 2023-11-02T08:37:27.036 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2023-11-02T08:37:27.036 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2023-11-02T08:37:27.036 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2023-11-02T08:37:27.036 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.036 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbd9ff4640 4 set_mon_vals no callback set 2023-11-02T08:37:27.037 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 5 monclient: authenticate success, global_id 4377 2023-11-02T08:37:27.037 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 10 monclient: _renew_subs 2023-11-02T08:37:27.037 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:37:27.037 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55776a388340 con 0x55776a45f5e0 2023-11-02T08:37:27.037 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:37:27.037 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 10 monclient: _renew_subs 2023-11-02T08:37:27.038 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:37:27.038 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55776a37b780 con 0x55776a45f5e0 2023-11-02T08:37:27.038 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 1 librados: init done 2023-11-02T08:37:27.039 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.131+0000 7fcbde4f9780 10 librados: wait_for_osdmap waiting 2023-11-02T08:37:27.039 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.233+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== mon.0 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcbd4011c70 con 0x55776a45f5e0 2023-11-02T08:37:27.039 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.234+0000 7fcbd97f3640 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fcbc806a000 0x7fcbc806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:37:27.044 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_api_cls_remote_reads 2023-11-02T08:37:27.044 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:27.044 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG_backup=en_US.UTF-8 2023-11-02T08:37:27.044 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:27.045 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /etc/locale.conf ']' 2023-11-02T08:37:27.045 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -x /usr/bin/sed ']' 2023-11-02T08:37:27.045 INFO:tasks.workunit.client.0.smithi151.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-11-02T08:37:27.045 INFO:tasks.workunit.client.0.smithi151.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-11-02T08:37:27.045 INFO:tasks.workunit.client.0.smithi151.stderr:++++ export LANG=en_US.UTF-8 2023-11-02T08:37:27.046 INFO:tasks.workunit.client.0.smithi151.stderr:++++ LANG=en_US.UTF-8 2023-11-02T08:37:27.046 INFO:tasks.workunit.client.0.smithi151.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-11-02T08:37:27.046 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-11-02T08:37:27.046 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:27.046 INFO:tasks.workunit.client.0.smithi151.stderr:+++ LANG=en_US.UTF-8 2023-11-02T08:37:27.047 INFO:tasks.workunit.client.0.smithi151.stderr:+++ unset LANG_backup config 2023-11-02T08:37:27.047 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n '' ']' 2023-11-02T08:37:27.047 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-11-02T08:37:27.047 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' dumb = linux ']' 2023-11-02T08:37:27.047 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.048 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-11-02T08:37:27.048 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.048 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/less.sh 2023-11-02T08:37:27.048 INFO:tasks.workunit.client.0.smithi151.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-11-02T08:37:27.048 INFO:tasks.workunit.client.0.smithi151.stderr:++ for i in /etc/profile.d/*.sh 2023-11-02T08:37:27.049 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-11-02T08:37:27.049 INFO:tasks.workunit.client.0.smithi151.stderr:++ '[' '' ']' 2023-11-02T08:37:27.049 INFO:tasks.workunit.client.0.smithi151.stderr:++ . /etc/profile.d/which2.sh 2023-11-02T08:37:27.049 INFO:tasks.workunit.client.0.smithi151.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-11-02T08:37:27.049 INFO:tasks.workunit.client.0.smithi151.stderr:+++++ readlink /proc/92006/exe 2023-11-02T08:37:27.050 INFO:tasks.workunit.client.0.smithi151.stderr:++++ basename /usr/bin/bash 2023-11-02T08:37:27.050 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_declare='declare -f' 2023-11-02T08:37:27.050 INFO:tasks.workunit.client.0.smithi151.stderr:+++ which_opt=-f 2023-11-02T08:37:27.050 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export which_declare 2023-11-02T08:37:27.050 INFO:tasks.workunit.client.0.smithi151.stderr:+++ export -f which 2023-11-02T08:37:27.051 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset i 2023-11-02T08:37:27.051 INFO:tasks.workunit.client.0.smithi151.stderr:++ unset -f pathmunge 2023-11-02T08:37:27.051 INFO:tasks.workunit.client.0.smithi151.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-11-02T08:37:27.051 INFO:tasks.workunit.client.0.smithi151.stderr:+ export PATH 2023-11-02T08:37:27.051 INFO:tasks.workunit.client.0.smithi151.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-11-02T08:37:27.052 INFO:tasks.workunit.client.0.smithi151.stderr:+ unset rc 2023-11-02T08:37:27.052 INFO:tasks.workunit.client.0.smithi151.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-11-02T08:37:27.052 INFO:tasks.workunit.client.0.smithi151.stderr:+ ceph_test_rados_open_pools_parallel 2023-11-02T08:37:27.052 INFO:tasks.workunit.client.0.smithi151.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-11-02T08:37:27.147 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.234+0000 7fcbdb7f7640 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fcbc806a000 0x7fcbc806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:37:27.148 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.234+0000 7fcbdb7f7640 10 monclient: get_auth_request con 0x7fcbc806a000 auth_method 0 2023-11-02T08:37:27.148 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.235+0000 7fcbdb7f7640 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fcbc806a000 0x7fcbc806c4c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55776a4e6a50 tx=0x7fcbd0005530 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:37:27.148 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.235+0000 7fcbd97f3640 10 client.4377.objecter ms_handle_connect 0x7fcbc806a000 2023-11-02T08:37:27.148 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.277+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== mon.0 v2:172.21.15.151:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcbd4056e70 con 0x55776a45f5e0 2023-11-02T08:37:27.148 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.277+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(22..22 src has 1..22) v4 2023-11-02T08:37:27.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.277+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [22,22] > 0 2023-11-02T08:37:27.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.277+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding full epoch 22 2023-11-02T08:37:27.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.278+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:27.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.278+0000 7fcbde4f9780 10 librados: wait_for_osdmap done waiting 2023-11-02T08:37:27.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.278+0000 7fcbde4f9780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:37:27.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.278+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcbcc0587d0 con 0x55776a45f5e0 2023-11-02T08:37:27.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.368+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== mon.0 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbd4056540 con 0x55776a45f5e0 2023-11-02T08:37:27.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.368+0000 7fcbd97f3640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-11-02T08:37:27.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.368+0000 7fcbda7f5640 10 client.4377.objecter _get_latest_version latest 22, have it 2023-11-02T08:37:27.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.369+0000 7fcbde4f9780 10 client.4377.objecter create_pool name=ceph_test_rados_api_asio 2023-11-02T08:37:27.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.369+0000 7fcbde4f9780 10 client.4377.objecter pool_op_submit 1 2023-11-02T08:37:27.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.369+0000 7fcbde4f9780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:37:27.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:26.369+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55776a3e7780 con 0x55776a45f5e0 2023-11-02T08:37:27.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== mon.0 v2:172.21.15.151:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v23) v1 ==== 43+0+0 (secure 0 0 0) 0x7fcbd401f020 con 0x55776a45f5e0 2023-11-02T08:37:27.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 pool_op_reply(tid 1 (0) Success v23) v1 2023-11-02T08:37:27.152 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 10 client.4377.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v23) v1 2023-11-02T08:37:27.152 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 10 client.4377.objecter have request 1 at 0x55776a4277c0 Op: create 2023-11-02T08:37:27.152 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 20 client.4377.objecter waiting for client to reach epoch 23 before calling back 2023-11-02T08:37:27.797 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-11-02T08:37:27.797 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 10 monclient: _renew_subs 2023-11-02T08:37:27.798 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.151:3300/0 2023-11-02T08:37:27.798 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=23}) v3 -- 0x7fcbc806fd40 con 0x55776a45f5e0 2023-11-02T08:37:27.798 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.144+0000 7fcbd97f3640 10 client.4377.objecter done 2023-11-02T08:37:27.798 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== mon.0 v2:172.21.15.151:3300/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 3231+0+0 (secure 0 0 0) 0x7fcbd4056770 con 0x55776a45f5e0 2023-11-02T08:37:27.798 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(23..23 src has 1..23) v4 2023-11-02T08:37:27.799 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [23,23] > 22 2023-11-02T08:37:27.801 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 23 2023-11-02T08:37:27.802 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:27.802 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 10 librados: set snap read head -> 1 2023-11-02T08:37:27.802 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 10 librados: writefull oid=exist nspace= 2023-11-02T08:37:27.802 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a42aea0 2023-11-02T08:37:27.802 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 23 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:27.803 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:27.803 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:27.803 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x55776a4ea290 0x55776a4e2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T08:37:27.803 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:27.803 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [writefull 0~5 in=5b] tid 2 osd.4 2023-11-02T08:37:27.804 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:27.804 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 2 2023-11-02T08:37:27.804 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 15 client.4377.objecter _send_op 2 to 10.1 on osd.4 2023-11-02T08:37:27.804 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:2 10.1 10:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) v8 -- 0x55776a42b8a0 con 0x55776a4ea290 2023-11-02T08:37:27.804 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:27.805 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.222+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:27.805 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.758+0000 7fcbdc7f9640 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x55776a4ea290 0x55776a4e2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-02T08:37:36.901 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.794+0000 7fcbdc7f9640 10 monclient: get_auth_request con 0x55776a4ea290 auth_method 0 2023-11-02T08:37:36.901 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.925+0000 7fcbd97f3640 10 client.4377.objecter ms_handle_connect 0x55776a4ea290 2023-11-02T08:37:36.901 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:27.925+0000 7fcbdc7f9640 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x55776a4ea290 0x55776a4e2010 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-02T08:37:36.902 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:29.035+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:36.902 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:29.035+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:36.902 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:29.035+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:36.902 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:29.035+0000 7fcbc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:37:36.902 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:30.035+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:36.903 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:30.035+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:36.903 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:30.035+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:36.903 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:31.035+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:36.903 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:31.035+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:36.903 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:31.035+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:36.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:31.132+0000 7fcbde25c640 10 client.4377.objecter tick 2023-11-02T08:37:36.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:32.035+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:36.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:32.035+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:36.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:32.035+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:36.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:33.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:36.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:33.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:36.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:33.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:36.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:34.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:36.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:34.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:36.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:34.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:36.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:35.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:36.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:35.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:36.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:35.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:36.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:36.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:36.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:36.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.133+0000 7fcbde25c640 10 client.4377.objecter tick 2023-11-02T08:37:36.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.897+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 1 ==== osd_map(24..28 src has 1..28) v4 ==== 19231+0+0 (crc 0 0 0) 0x7fcbcc011070 con 0x55776a4ea290 2023-11-02T08:37:36.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.897+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(24..28 src has 1..28) v4 2023-11-02T08:37:36.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.897+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [24,28] > 23 2023-11-02T08:37:39.159 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.897+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 24 2023-11-02T08:37:39.160 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.897+0000 7fcbd97f3640 10 client.4377.objecter checking op 2 2023-11-02T08:37:39.160 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.897+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 24 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:39.160 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.897+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.160 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.897+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 25 2023-11-02T08:37:39.160 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 10 client.4377.objecter checking op 2 2023-11-02T08:37:39.161 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 25 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:39.161 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.161 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 26 2023-11-02T08:37:39.161 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 10 client.4377.objecter checking op 2 2023-11-02T08:37:39.161 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 26 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:39.162 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.162 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 27 2023-11-02T08:37:39.162 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 10 client.4377.objecter checking op 2 2023-11-02T08:37:39.162 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 27 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:39.162 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.163 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 28 2023-11-02T08:37:39.163 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 10 client.4377.objecter checking op 2 2023-11-02T08:37:39.163 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 28 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:39.163 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.163 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:39.164 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:36.898+0000 7fcbd97f3640 20 client.4377.objecter 2 10.d3a59201 osd.4 exist [writefull 0~5 in=5b] 2023-11-02T08:37:39.164 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:37.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:39.164 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:37.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:39.164 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:37.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:39.164 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:38.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:39.164 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:38.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:39.165 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:38.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:39.165 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:39.165 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:39.165 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:39.165 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.036+0000 7fcbc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-11-02T08:37:39.166 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 2 ==== osd_op_reply(2 exist [writefull 0~5] v28'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:39.166 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(2 exist [writefull 0~5] v28'1 uv1 ondisk = 0) v8 2023-11-02T08:37:39.166 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:39.166 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 2 ondisk uv 1 in 10.1 attempt 0 2023-11-02T08:37:39.167 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:39.167 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 2 2023-11-02T08:37:39.167 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 2 2023-11-02T08:37:39.167 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.167 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 2 2023-11-02T08:37:39.168 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:39.168 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-11-02T08:37:39.168 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 10 librados: Objecter returned from writefull r=0 2023-11-02T08:37:39.169 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a42aea0 2023-11-02T08:37:39.169 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 28 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.169 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.169 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:39.170 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:39.170 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [read 0~256] tid 3 osd.4 2023-11-02T08:37:39.170 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:39.170 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 3 2023-11-02T08:37:39.171 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 15 client.4377.objecter _send_op 3 to 10.1 on osd.4 2023-11-02T08:37:39.171 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:3 10.1 10:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x55776a4e2550 con 0x55776a4ea290 2023-11-02T08:37:39.171 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.171 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:39.172 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a4e2950 2023-11-02T08:37:39.172 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 28 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.172 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:39.172 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:39.172 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.266 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid noexist '@10' '@10' [read 0~256] tid 4 osd.4 2023-11-02T08:37:39.266 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.267 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 4 2023-11-02T08:37:39.267 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 15 client.4377.objecter _send_op 4 to 10.1 on osd.4 2023-11-02T08:37:39.267 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:4 10.1 10:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28) v8 -- 0x55776a4e2e40 con 0x55776a4ea290 2023-11-02T08:37:39.267 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:39.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.156+0000 7fcbde4f9780 5 client.4377.objecter 2 in flight 2023-11-02T08:37:39.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 3 ==== osd_map(29..30 src has 1..30) v4 ==== 1623+0+0 (crc 0 0 0) 0x7fcbcc0584f0 con 0x55776a4ea290 2023-11-02T08:37:39.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(29..30 src has 1..30) v4 2023-11-02T08:37:39.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [29,30] > 28 2023-11-02T08:37:39.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 29 2023-11-02T08:37:39.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 10 client.4377.objecter checking op 3 2023-11-02T08:37:39.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 29 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 10 client.4377.objecter checking op 4 2023-11-02T08:37:39.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 29 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:39.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 30 2023-11-02T08:37:39.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 10 client.4377.objecter checking op 3 2023-11-02T08:37:39.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 30 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.271 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 10 client.4377.objecter checking op 4 2023-11-02T08:37:39.273 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 30 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.273 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:39.273 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:39.273 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter 3 10.d3a59201 osd.4 exist [read 0~256] 2023-11-02T08:37:39.274 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.197+0000 7fcbd97f3640 20 client.4377.objecter 4 10.a7bbaa1 osd.4 noexist [read 0~256] 2023-11-02T08:37:39.274 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:39.274 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-11-02T08:37:39.274 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:39.274 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 3 ondisk uv 1 in 10.1 attempt 0 2023-11-02T08:37:39.275 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 5 2023-11-02T08:37:39.275 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 3 2023-11-02T08:37:39.275 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 3 2023-11-02T08:37:39.275 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:39.275 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 3 2023-11-02T08:37:39.276 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 5 client.4377.objecter 1 in flight 2023-11-02T08:37:39.276 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (100 ms) 2023-11-02T08:37:39.276 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-11-02T08:37:39.276 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:39.276 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-02T08:37:39.277 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:39.277 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 4 ondisk uv 0 in 10.1 attempt 0 2023-11-02T08:37:39.277 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 10 client.4377.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-02T08:37:39.277 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:39.277 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 4 2023-11-02T08:37:39.278 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 4 2023-11-02T08:37:39.278 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.278 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 4 2023-11-02T08:37:39.278 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.255+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:39.278 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a42aea0 2023-11-02T08:37:39.279 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 30 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.279 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.282 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:39.283 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:39.283 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [read 0~256] tid 5 osd.4 2023-11-02T08:37:39.284 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:39.318 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 5 2023-11-02T08:37:39.319 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 15 client.4377.objecter _send_op 5 to 10.1 on osd.4 2023-11-02T08:37:39.320 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:5 10.1 10:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55776a4e2900 con 0x55776a4ea290 2023-11-02T08:37:39.320 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.321 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:39.322 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a58f7f0 2023-11-02T08:37:39.323 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 30 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.323 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:39.324 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:39.324 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.324 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid noexist '@10' '@10' [read 0~256] tid 6 osd.4 2023-11-02T08:37:39.325 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.326 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 6 2023-11-02T08:37:39.327 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 15 client.4377.objecter _send_op 6 to 10.1 on osd.4 2023-11-02T08:37:39.328 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:6 10.1 10:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55776a58fce0 con 0x55776a4ea290 2023-11-02T08:37:39.328 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:39.329 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.256+0000 7fcbde4f9780 5 client.4377.objecter 2 in flight 2023-11-02T08:37:39.329 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 6 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:39.330 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-11-02T08:37:39.331 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:39.331 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 5 ondisk uv 1 in 10.1 attempt 0 2023-11-02T08:37:39.333 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 5 2023-11-02T08:37:39.333 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 5 2023-11-02T08:37:39.333 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 5 2023-11-02T08:37:39.334 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:39.334 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 5 2023-11-02T08:37:39.334 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 5 client.4377.objecter 1 in flight 2023-11-02T08:37:39.334 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 7 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:39.335 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-02T08:37:39.335 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:39.335 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 6 ondisk uv 0 in 10.1 attempt 0 2023-11-02T08:37:39.335 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 10 client.4377.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-02T08:37:39.335 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:39.336 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 6 2023-11-02T08:37:39.337 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 6 2023-11-02T08:37:39.337 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.339 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 6 2023-11-02T08:37:39.339 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:39.339 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (53 ms) 2023-11-02T08:37:39.339 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-11-02T08:37:39.340 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a58f7f0 2023-11-02T08:37:39.340 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 30 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.340 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:39.340 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:39.340 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:39.341 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [read 0~256] tid 7 osd.4 2023-11-02T08:37:39.341 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:39.341 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 7 2023-11-02T08:37:39.341 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 15 client.4377.objecter _send_op 7 to 10.1 on osd.4 2023-11-02T08:37:39.342 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:7 10.1 10:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55776a42aea0 con 0x55776a4ea290 2023-11-02T08:37:39.342 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.342 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:39.343 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5eb2a0 2023-11-02T08:37:39.343 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 30 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:39.343 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:39.343 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:39.346 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.347 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid noexist '@10' '@10' [read 0~256] tid 8 osd.4 2023-11-02T08:37:39.347 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.347 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 8 2023-11-02T08:37:39.348 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 15 client.4377.objecter _send_op 8 to 10.1 on osd.4 2023-11-02T08:37:39.348 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:8 10.1 10:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55776a5eb790 con 0x55776a4ea290 2023-11-02T08:37:39.348 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:39.348 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.309+0000 7fcbde4f9780 5 client.4377.objecter 2 in flight 2023-11-02T08:37:39.349 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 8 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:39.349 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-11-02T08:37:39.349 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:39.349 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 7 ondisk uv 1 in 10.1 attempt 0 2023-11-02T08:37:39.349 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 5 2023-11-02T08:37:39.350 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 7 2023-11-02T08:37:39.350 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 7 2023-11-02T08:37:39.350 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:39.350 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 7 2023-11-02T08:37:39.350 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 5 client.4377.objecter 1 in flight 2023-11-02T08:37:39.351 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 9 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:39.351 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-02T08:37:39.351 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:39.351 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 8 ondisk uv 0 in 10.1 attempt 0 2023-11-02T08:37:39.352 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 10 client.4377.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-02T08:37:39.352 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:39.352 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 8 2023-11-02T08:37:39.352 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 8 2023-11-02T08:37:39.353 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:39.785 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:02:20.675 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:39.853 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:02:20.743 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:39.857 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:20.733 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:39.921 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:02:20.811 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:39.925 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:02:20.816 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:39.938 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:20.813 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:40.036 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:02:20.927 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:40.080 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:20.955 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:40.096 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:02:20.986 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:41.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 8 2023-11-02T08:37:41.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:41.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (31 ms) 2023-11-02T08:37:41.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-11-02T08:37:41.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbde4f9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-11-02T08:37:41.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbde4f9780 20 librados: queue_aio_write 0x55776a3d6dc0 completion 0x55776a4e6510 write_seq 1 2023-11-02T08:37:41.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a58f7f0 2023-11-02T08:37:41.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 30 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:41.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:41.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:41.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.340+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:41.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.341+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [write 0~5 in=5b] tid 9 osd.4 2023-11-02T08:37:41.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.341+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:41.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.341+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 9 2023-11-02T08:37:41.908 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.341+0000 7fcbde4f9780 15 client.4377.objecter _send_op 9 to 10.1 on osd.4 2023-11-02T08:37:41.908 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.341+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:9 10.1 10:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55776a590110 con 0x55776a4ea290 2023-11-02T08:37:41.908 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.341+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:41.908 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.341+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:41.908 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:39.341+0000 7fcbde4f9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-11-02T08:37:41.909 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:40.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:41.909 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:40.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:41.909 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:40.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:41.909 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:41.910 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:41.910 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:41.910 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.134+0000 7fcbde25c640 10 client.4377.objecter tick 2023-11-02T08:37:41.910 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 10 ==== osd_op_reply(9 exist [write 0~5] v30'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:41.910 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(9 exist [write 0~5] v30'2 uv2 ondisk = 0) v8 2023-11-02T08:37:41.911 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:41.911 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 9 ondisk uv 2 in 10.1 attempt 0 2023-11-02T08:37:41.911 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:41.976 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 9 2023-11-02T08:37:41.976 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 9 2023-11-02T08:37:41.976 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:41.976 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 9 2023-11-02T08:37:41.977 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:41.977 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbdc7f9640 20 librados: complete_aio_write 0x55776a4e6510 2023-11-02T08:37:41.977 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2560 ms) 2023-11-02T08:37:41.977 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-11-02T08:37:41.977 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbde4f9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-11-02T08:37:41.978 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbde4f9780 20 librados: queue_aio_write 0x55776a3d6dc0 completion 0x55776a5904c0 write_seq 2 2023-11-02T08:37:41.978 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a58f7f0 2023-11-02T08:37:41.978 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.900+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 30 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:41.978 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:41.978 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:41.979 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:41.979 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [write 0~5 in=5b] tid 10 osd.4 2023-11-02T08:37:41.979 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:41.979 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 10 2023-11-02T08:37:41.979 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 15 client.4377.objecter _send_op 10 to 10.1 on osd.4 2023-11-02T08:37:41.980 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:10 10.1 10:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55776a590110 con 0x55776a4ea290 2023-11-02T08:37:41.980 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:41.980 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:41.980 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.901+0000 7fcbde4f9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-11-02T08:37:41.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.970+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 11 ==== osd_map(31..31 src has 1..31) v4 ==== 1023+0+0 (crc 0 0 0) 0x7fcbcc063440 con 0x55776a4ea290 2023-11-02T08:37:41.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.970+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(31..31 src has 1..31) v4 2023-11-02T08:37:41.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.970+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [31,31] > 30 2023-11-02T08:37:41.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.970+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 31 2023-11-02T08:37:41.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.970+0000 7fcbd97f3640 10 client.4377.objecter checking op 10 2023-11-02T08:37:41.982 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.970+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 31 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:42.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.970+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:42.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.970+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:42.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:41.970+0000 7fcbd97f3640 20 client.4377.objecter 10 10.d3a59201 osd.4 exist [write 0~5 in=5b] 2023-11-02T08:37:42.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:42.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:42.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:42.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 12 ==== osd_op_reply(10 exist [write 0~5] v31'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:42.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(10 exist [write 0~5] v31'3 uv3 ondisk = 0) v8 2023-11-02T08:37:42.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:42.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 10 ondisk uv 3 in 10.1 attempt 0 2023-11-02T08:37:42.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:42.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 10 2023-11-02T08:37:42.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 10 2023-11-02T08:37:42.572 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:42.572 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 10 2023-11-02T08:37:42.572 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:42.572 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbdc7f9640 20 librados: complete_aio_write 0x55776a5904c0 2023-11-02T08:37:42.572 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (664 ms) 2023-11-02T08:37:42.573 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-11-02T08:37:42.573 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-11-02T08:37:42.573 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 20 librados: queue_aio_write 0x55776a3d6dc0 completion 0x55776a4e6280 write_seq 3 2023-11-02T08:37:42.573 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a58f7f0 2023-11-02T08:37:42.574 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 31 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:42.574 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:42.574 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:42.574 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:42.574 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [write 0~5 in=5b] tid 11 osd.4 2023-11-02T08:37:42.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:42.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 11 2023-11-02T08:37:42.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 15 client.4377.objecter _send_op 11 to 10.1 on osd.4 2023-11-02T08:37:43.565 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:11 10.1 10:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) v8 -- 0x55776a5ce650 con 0x55776a4ea290 2023-11-02T08:37:43.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:43.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:43.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.565+0000 7fcbde4f9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-11-02T08:37:43.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.803+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 13 ==== osd_map(32..32 src has 1..33) v4 ==== 652+0+0 (crc 0 0 0) 0x7fcbcc058da0 con 0x55776a4ea290 2023-11-02T08:37:43.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.803+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(32..32 src has 1..33) v4 2023-11-02T08:37:43.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.803+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [32,32] > 31 2023-11-02T08:37:43.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.803+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 32 2023-11-02T08:37:43.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.803+0000 7fcbd97f3640 10 client.4377.objecter checking op 11 2023-11-02T08:37:43.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.803+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 32 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:43.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.803+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:43.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.803+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:43.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:42.803+0000 7fcbd97f3640 20 client.4377.objecter 11 10.d3a59201 osd.4 exist [write 0~5 in=5b] 2023-11-02T08:37:43.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:43.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.036+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:43.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.036+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:43.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 14 ==== osd_op_reply(11 exist [write 0~5] v32'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:43.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(11 exist [write 0~5] v32'4 uv4 ondisk = 0) v8 2023-11-02T08:37:43.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:43.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 11 ondisk uv 4 in 10.1 attempt 0 2023-11-02T08:37:43.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:43.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 11 2023-11-02T08:37:43.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 11 2023-11-02T08:37:43.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:43.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 11 2023-11-02T08:37:43.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:43.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbdc7f9640 20 librados: complete_aio_write 0x55776a4e6280 2023-11-02T08:37:43.573 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (997 ms) 2023-11-02T08:37:43.573 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-11-02T08:37:43.574 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5ab280 2023-11-02T08:37:43.574 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 32 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:44.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:44.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:44.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [read 0~0] tid 12 osd.4 2023-11-02T08:37:44.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:44.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 12 2023-11-02T08:37:44.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 15 client.4377.objecter _send_op 12 to 10.1 on osd.4 2023-11-02T08:37:44.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:12 10.1 10:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55776a5ab770 con 0x55776a4ea290 2023-11-02T08:37:44.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:44.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5abc30 2023-11-02T08:37:44.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 32 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:44.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:44.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid noexist '@10' '@10' [read 0~0] tid 13 osd.4 2023-11-02T08:37:44.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 13 2023-11-02T08:37:44.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 15 client.4377.objecter _send_op 13 to 10.1 on osd.4 2023-11-02T08:37:44.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:13 10.1 10:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55776a5ceab0 con 0x55776a4ea290 2023-11-02T08:37:44.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:44.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:43.562+0000 7fcbde4f9780 5 client.4377.objecter 2 in flight 2023-11-02T08:37:44.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.036+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:44.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.037+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:44.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.037+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:44.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 15 ==== osd_map(33..33 src has 1..34) v4 ==== 697+0+0 (crc 0 0 0) 0x7fcbcc063920 con 0x55776a4ea290 2023-11-02T08:37:44.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 16 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:44.134 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(33..33 src has 1..34) v4 2023-11-02T08:37:44.134 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-11-02T08:37:44.134 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:44.134 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [33,33] > 32 2023-11-02T08:37:44.135 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 33 2023-11-02T08:37:44.135 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 10 client.4377.objecter checking op 12 2023-11-02T08:37:44.136 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 33 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.136 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:44.136 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 10 client.4377.objecter checking op 13 2023-11-02T08:37:44.137 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 33 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.137 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:44.137 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:44.137 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 20 client.4377.objecter 12 10.d3a59201 osd.4 exist [read 0~0] 2023-11-02T08:37:44.137 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbd97f3640 20 client.4377.objecter 13 10.a7bbaa1 osd.4 noexist [read 0~0] 2023-11-02T08:37:44.138 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 12 ondisk uv 4 in 10.1 attempt 0 2023-11-02T08:37:44.138 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 5 2023-11-02T08:37:44.138 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 12 2023-11-02T08:37:44.138 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 12 2023-11-02T08:37:44.138 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:44.139 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 12 2023-11-02T08:37:44.139 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 5 client.4377.objecter 1 in flight 2023-11-02T08:37:44.139 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 17 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:44.139 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-02T08:37:44.140 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:44.140 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 13 ondisk uv 0 in 10.1 attempt 0 2023-11-02T08:37:44.140 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 10 client.4377.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-02T08:37:44.140 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:44.141 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 13 2023-11-02T08:37:44.141 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 13 2023-11-02T08:37:44.141 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.141 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 13 2023-11-02T08:37:44.142 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:44.142 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (561 ms) 2023-11-02T08:37:44.142 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-11-02T08:37:44.142 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5abc30 2023-11-02T08:37:44.143 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 33 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.143 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:44.143 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:44.146 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:44.147 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [read 0~0] tid 14 osd.4 2023-11-02T08:37:44.147 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:44.147 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 14 2023-11-02T08:37:44.148 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 15 client.4377.objecter _send_op 14 to 10.1 on osd.4 2023-11-02T08:37:44.148 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:14 10.1 10:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55776a5ab280 con 0x55776a4ea290 2023-11-02T08:37:44.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:44.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5ceab0 2023-11-02T08:37:44.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 33 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:44.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:44.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid noexist '@10' '@10' [read 0~0] tid 15 osd.4 2023-11-02T08:37:44.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 15 2023-11-02T08:37:44.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 15 client.4377.objecter _send_op 15 to 10.1 on osd.4 2023-11-02T08:37:44.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:15 10.1 10:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55776a5cefa0 con 0x55776a4ea290 2023-11-02T08:37:44.152 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:44.539 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.123+0000 7fcbde4f9780 5 client.4377.objecter 2 in flight 2023-11-02T08:37:44.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 18 ==== osd_map(34..34 src has 1..34) v4 ==== 1386+0+0 (crc 0 0 0) 0x7fcbcc063ae0 con 0x55776a4ea290 2023-11-02T08:37:44.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 19 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:44.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(34..34 src has 1..34) v4 2023-11-02T08:37:44.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-11-02T08:37:44.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:44.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [34,34] > 33 2023-11-02T08:37:44.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 34 2023-11-02T08:37:44.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 10 client.4377.objecter checking op 14 2023-11-02T08:37:44.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 34 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.568 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:44.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 10 client.4377.objecter checking op 15 2023-11-02T08:37:44.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 34 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:44.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:44.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 20 client.4377.objecter 14 10.d3a59201 osd.4 exist [read 0~0] 2023-11-02T08:37:44.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbd97f3640 20 client.4377.objecter 15 10.a7bbaa1 osd.4 noexist [read 0~0] 2023-11-02T08:37:44.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 14 ondisk uv 4 in 10.1 attempt 0 2023-11-02T08:37:44.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 5 2023-11-02T08:37:44.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 14 2023-11-02T08:37:44.570 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 14 2023-11-02T08:37:44.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:44.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 14 2023-11-02T08:37:44.571 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 5 client.4377.objecter 1 in flight 2023-11-02T08:37:44.572 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 20 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:44.572 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-02T08:37:44.572 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:44.573 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 15 ondisk uv 0 in 10.1 attempt 0 2023-11-02T08:37:44.573 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 10 client.4377.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-02T08:37:44.573 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:44.574 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 15 2023-11-02T08:37:44.574 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 15 2023-11-02T08:37:44.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 15 2023-11-02T08:37:44.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:44.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (412 ms) 2023-11-02T08:37:44.575 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-11-02T08:37:44.576 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5abc30 2023-11-02T08:37:44.576 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 34 base exist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.576 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:44.580 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:44.580 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:44.580 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [read 0~0] tid 16 osd.4 2023-11-02T08:37:44.580 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:44.580 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 16 2023-11-02T08:37:44.581 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 15 client.4377.objecter _send_op 16 to 10.1 on osd.4 2023-11-02T08:37:44.581 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:16 10.1 10:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x55776a5ab280 con 0x55776a4ea290 2023-11-02T08:37:44.581 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.581 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:44.581 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5cdfe0 2023-11-02T08:37:44.582 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 34 base noexist @10 precalc_pgid 0 pgid 0.0 is_read 2023-11-02T08:37:44.583 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target noexist @10 -> pgid 10.a7bbaa1 2023-11-02T08:37:44.583 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.a7bbaa1 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:44.583 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.583 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid noexist '@10' '@10' [read 0~0] tid 17 osd.4 2023-11-02T08:37:44.584 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.584 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 17 2023-11-02T08:37:44.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 15 client.4377.objecter _send_op 17 to 10.1 on osd.4 2023-11-02T08:37:44.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:17 10.1 10:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x55776a5ce4d0 con 0x55776a4ea290 2023-11-02T08:37:44.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:44.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.535+0000 7fcbde4f9780 5 client.4377.objecter 2 in flight 2023-11-02T08:37:44.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 21 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:44.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-11-02T08:37:44.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:44.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 16 ondisk uv 4 in 10.1 attempt 0 2023-11-02T08:37:44.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 5 2023-11-02T08:37:44.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 16 2023-11-02T08:37:44.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 16 2023-11-02T08:37:44.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 5 2023-11-02T08:37:44.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 16 2023-11-02T08:37:44.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 5 client.4377.objecter 1 in flight 2023-11-02T08:37:44.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 22 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:44.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-11-02T08:37:44.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:44.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 17 ondisk uv 0 in 10.1 attempt 0 2023-11-02T08:37:44.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 10 client.4377.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-11-02T08:37:44.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:44.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 17 2023-11-02T08:37:44.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 17 2023-11-02T08:37:44.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:44.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 17 2023-11-02T08:37:44.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:44.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (390 ms) 2023-11-02T08:37:44.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-11-02T08:37:44.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 20 librados: queue_aio_write 0x55776a3d6dc0 completion 0x55776a5ab8f0 write_seq 4 2023-11-02T08:37:44.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5abc30 2023-11-02T08:37:46.914 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 34 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:46.915 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:46.915 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:46.915 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:46.915 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [writefull 0~5 in=5b] tid 18 osd.4 2023-11-02T08:37:46.915 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:46.916 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 18 2023-11-02T08:37:46.916 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 15 client.4377.objecter _send_op 18 to 10.1 on osd.4 2023-11-02T08:37:46.916 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:18 10.1 10:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) v8 -- 0x55776a5ab280 con 0x55776a4ea290 2023-11-02T08:37:46.916 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:46.916 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:44.925+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:46.917 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.037+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:46.917 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.037+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:46.917 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.037+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:46.917 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.605+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 23 ==== osd_map(35..35 src has 1..35) v4 ==== 1020+0+0 (crc 0 0 0) 0x7fcbcc007040 con 0x55776a4ea290 2023-11-02T08:37:46.917 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.605+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(35..35 src has 1..35) v4 2023-11-02T08:37:46.918 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.605+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [35,35] > 34 2023-11-02T08:37:46.918 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.605+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 35 2023-11-02T08:37:46.918 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.605+0000 7fcbd97f3640 10 client.4377.objecter checking op 18 2023-11-02T08:37:46.918 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.605+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 35 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:46.918 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.605+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:46.919 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.605+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:46.919 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:45.605+0000 7fcbd97f3640 20 client.4377.objecter 18 10.d3a59201 osd.4 exist [writefull 0~5 in=5b] 2023-11-02T08:37:46.919 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.037+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:46.919 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.037+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:46.919 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.037+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:46.920 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.135+0000 7fcbde25c640 10 client.4377.objecter tick 2023-11-02T08:37:46.920 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 24 ==== osd_op_reply(18 exist [writefull 0~5] v35'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:47.249 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(18 exist [writefull 0~5] v35'5 uv5 ondisk = 0) v8 2023-11-02T08:37:47.249 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:47.249 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 18 ondisk uv 5 in 10.1 attempt 0 2023-11-02T08:37:47.249 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:47.250 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 18 2023-11-02T08:37:47.250 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 18 2023-11-02T08:37:47.250 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:47.250 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 18 2023-11-02T08:37:47.250 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:47.251 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbdc7f9640 20 librados: complete_aio_write 0x55776a5ab8f0 2023-11-02T08:37:47.251 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1986 ms) 2023-11-02T08:37:47.251 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-11-02T08:37:47.251 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 20 librados: queue_aio_write 0x55776a3d6dc0 completion 0x55776a5ac120 write_seq 5 2023-11-02T08:37:47.252 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5abc30 2023-11-02T08:37:47.252 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 35 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:47.252 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:47.252 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:47.252 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:47.253 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [writefull 0~5 in=5b] tid 19 osd.4 2023-11-02T08:37:47.253 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:47.253 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 19 2023-11-02T08:37:47.253 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 15 client.4377.objecter _send_op 19 to 10.1 on osd.4 2023-11-02T08:37:47.253 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:19 10.1 10:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35) v8 -- 0x55776a5ab280 con 0x55776a4ea290 2023-11-02T08:37:47.254 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:47.254 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:46.911+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:47.254 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.037+0000 7fcbc3fff640 10 monclient: tick 2023-11-02T08:37:47.254 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.037+0000 7fcbc3fff640 10 monclient: _check_auth_tickets 2023-11-02T08:37:47.254 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.037+0000 7fcbc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-11-02T08:37:47.255 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 25 ==== osd_map(36..37 src has 1..37) v4 ==== 943+0+0 (crc 0 0 0) 0x7fcbcc007040 con 0x55776a4ea290 2023-11-02T08:37:47.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_map(36..37 src has 1..37) v4 2023-11-02T08:37:47.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map got epochs [36,37] > 35 2023-11-02T08:37:47.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 36 2023-11-02T08:37:47.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 10 client.4377.objecter checking op 19 2023-11-02T08:37:47.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 36 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:47.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:47.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 3 client.4377.objecter handle_osd_map decoding incremental epoch 37 2023-11-02T08:37:47.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 10 client.4377.objecter checking op 19 2023-11-02T08:37:47.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 20 client.4377.objecter _calc_target epoch 37 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:47.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:47.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 20 client.4377.objecter dump_active .. 0 homeless 2023-11-02T08:37:47.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.244+0000 7fcbd97f3640 20 client.4377.objecter 19 10.d3a59201 osd.4 exist [writefull 0~5 in=5b] 2023-11-02T08:37:47.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 26 ==== osd_op_reply(19 exist [writefull 0~5] v37'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:47.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(19 exist [writefull 0~5] v37'6 uv6 ondisk = 0) v8 2023-11-02T08:37:47.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:47.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 19 ondisk uv 6 in 10.1 attempt 0 2023-11-02T08:37:47.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:47.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 19 2023-11-02T08:37:47.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 19 2023-11-02T08:37:47.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:47.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 19 2023-11-02T08:37:47.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:47.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbdc7f9640 20 librados: complete_aio_write 0x55776a5ac120 2023-11-02T08:37:47.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (749 ms) 2023-11-02T08:37:47.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-11-02T08:37:47.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 20 librados: queue_aio_write 0x55776a3d6dc0 completion 0x55776a58fb50 write_seq 6 2023-11-02T08:37:47.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 10 client.4377.objecter _op_submit op 0x55776a5abc30 2023-11-02T08:37:47.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 20 client.4377.objecter _calc_target epoch 37 base exist @10 precalc_pgid 0 pgid 0.0 is_write 2023-11-02T08:37:47.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 20 client.4377.objecter _calc_target target exist @10 -> pgid 10.d3a59201 2023-11-02T08:37:47.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 10 client.4377.objecter _calc_target raw pgid 10.d3a59201 -> actual 10.1 acting [4,7] primary 4 2023-11-02T08:37:47.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 20 client.4377.objecter _get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:47.782 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 10 client.4377.objecter _op_submit oid exist '@10' '@10' [writefull 0~5 in=5b] tid 20 osd.4 2023-11-02T08:37:47.783 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 20 client.4377.objecter get_session s=0x55776a378d10 osd=4 3 2023-11-02T08:37:47.783 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 15 client.4377.objecter _session_op_assign 4 20 2023-11-02T08:37:47.783 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 15 client.4377.objecter _send_op 20 to 10.1 on osd.4 2023-11-02T08:37:47.783 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 --> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] -- osd_op(unknown.0.0:20 10.1 10:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) v8 -- 0x55776a5ab280 con 0x55776a4ea290 2023-11-02T08:37:47.784 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:47.784 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.660+0000 7fcbde4f9780 5 client.4377.objecter 1 in flight 2023-11-02T08:37:47.784 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 <== osd.4 v2:172.21.15.178:6810/86120 27 ==== osd_op_reply(20 exist [writefull 0~5] v37'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fcbcc060050 con 0x55776a4ea290 2023-11-02T08:37:47.784 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 10 client.4377.objecter ms_dispatch 0x55776a3cdb30 osd_op_reply(20 exist [writefull 0~5] v37'7 uv7 ondisk = 0) v8 2023-11-02T08:37:47.784 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 10 client.4377.objecter in handle_osd_op_reply 2023-11-02T08:37:47.784 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 7 client.4377.objecter handle_osd_op_reply 20 ondisk uv 7 in 10.1 attempt 0 2023-11-02T08:37:47.785 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 10 client.4377.objecter op 0 rval 0 len 0 2023-11-02T08:37:47.785 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 15 client.4377.objecter handle_osd_op_reply completed tid 20 2023-11-02T08:37:47.785 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 15 client.4377.objecter _finish_op 20 2023-11-02T08:37:47.785 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 4 2023-11-02T08:37:47.785 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 15 client.4377.objecter _session_op_remove 4 20 2023-11-02T08:37:47.786 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 5 client.4377.objecter 0 in flight 2023-11-02T08:37:47.786 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 20 librados: complete_aio_write 0x55776a58fb50 2023-11-02T08:37:47.786 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (119 ms) 2023-11-02T08:37:47.786 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 10 librados: watch_flush enter 2023-11-02T08:37:47.786 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 10 librados: watch_flush exit 2023-11-02T08:37:47.787 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 10 client.4377.objecter close_session for osd.4 2023-11-02T08:37:47.787 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x55776a4ea290 msgr2=0x55776a4e2010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:37:47.787 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6810/86120,v1:172.21.15.178:6811/86120] conn(0x55776a4ea290 0x55776a4e2010 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T08:37:47.787 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 20 client.4377.objecter put_session s=0x55776a378d10 osd=4 1 2023-11-02T08:37:47.787 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 20 client.4377.objecter shutdown clearing up homeless session... 2023-11-02T08:37:47.787 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 10 client.4377.objecter successfully canceled tick 2023-11-02T08:37:47.788 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fcbc806a000 msgr2=0x7fcbc806c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:37:47.788 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.178:6800/85498,v1:172.21.15.178:6801/85498] conn(0x7fcbc806a000 0x7fcbc806c4c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55776a4e6a50 tx=0x7fcbd0005530 comp rx=0 tx=0).stop 2023-11-02T08:37:47.788 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 10 monclient: shutdown 2023-11-02T08:37:47.789 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 20 monclient: shutdown discarding 0 pending message(s) 2023-11-02T08:37:47.789 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 msgr2=0x55776a58c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-02T08:37:47.789 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 --2- 172.21.15.151:0/4083040167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55776a45f5e0 0x55776a58c270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400d500 tx=0x7fcbd400bf90 comp rx=0 tx=0).stop 2023-11-02T08:37:47.789 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbdc7f9640 1 -- 172.21.15.151:0/4083040167 reap_dead start 2023-11-02T08:37:47.789 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 shutdown_connections 2023-11-02T08:37:47.790 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 >> 172.21.15.151:0/4083040167 conn(0x55776a424440 msgr2=0x55776a467f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-02T08:37:47.790 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 shutdown_connections 2023-11-02T08:37:47.790 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 -- 172.21.15.151:0/4083040167 wait complete. 2023-11-02T08:37:47.790 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.779+0000 7fcbde4f9780 1 librados: shutdown 2023-11-02T08:37:47.790 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [----------] 12 tests from AsioRados (8623 ms total) 2023-11-02T08:37:47.791 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: 2023-11-02T08:37:47.791 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [----------] Global test environment tear-down 2023-11-02T08:37:47.791 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (21777 ms total) 2023-11-02T08:37:47.791 INFO:tasks.workunit.client.0.smithi151.stdout: api_asio: [ PASSED ] 12 tests. 2023-11-02T08:37:54.831 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2023-11-02T08:37:54.831 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [----------] Global test environment set-up. 2023-11-02T08:37:54.831 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [----------] 4 tests from LibRadosService 2023-11-02T08:37:54.831 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-11-02T08:37:54.832 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5523 ms) 2023-11-02T08:37:54.832 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-11-02T08:37:54.832 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [ OK ] LibRadosService.RegisterLate (125 ms) 2023-11-02T08:37:54.832 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-11-02T08:37:54.832 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: cluster: 2023-11-02T08:37:54.833 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: id: 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:37:54.833 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: health: HEALTH_WARN 2023-11-02T08:37:54.833 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 5 pool(s) do not have an application enabled 2023-11-02T08:37:54.833 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 2023-11-02T08:37:54.833 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: services: 2023-11-02T08:37:54.834 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-11-02T08:37:54.834 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: mgr: x(active, since 115s), standbys: y 2023-11-02T08:37:54.834 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: osd: 8 osds: 8 up (since 73s), 8 in (since 114s) 2023-11-02T08:37:54.834 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-11-02T08:37:54.834 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 2023-11-02T08:37:54.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: data: 2023-11-02T08:37:54.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: pools: 25 pools, 745 pgs 2023-11-02T08:37:54.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: objects: 2 objects, 449 KiB 2023-11-02T08:37:54.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: usage: 213 MiB used, 720 GiB / 720 GiB avail 2023-11-02T08:37:54.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: pgs: 31.409% pgs unknown 2023-11-02T08:37:54.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 66.309% pgs not active 2023-11-02T08:37:54.836 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 436 creating+peering 2023-11-02T08:37:54.836 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 234 unknown 2023-11-02T08:37:54.836 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 58 creating+activating 2023-11-02T08:37:54.836 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 17 active+clean 2023-11-02T08:37:54.836 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 2023-11-02T08:37:54.837 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 2023-11-02T08:37:54.837 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: cluster: 2023-11-02T08:37:54.837 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: id: 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T08:37:54.837 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: health: HEALTH_WARN 2023-11-02T08:37:54.837 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 8 pool(s) do not have an application enabled 2023-11-02T08:37:54.838 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 2023-11-02T08:37:54.838 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: services: 2023-11-02T08:37:54.838 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-11-02T08:37:54.838 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: mgr: x(active, since 118s), standbys: y 2023-11-02T08:37:54.838 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: osd: 8 osds: 8 up (since 75s), 8 in (since 116s) 2023-11-02T08:37:54.839 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-11-02T08:37:54.839 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-11-02T08:37:54.839 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 2023-11-02T08:37:54.839 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: data: 2023-11-02T08:37:54.839 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: pools: 29 pools, 849 pgs 2023-11-02T08:37:54.839 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: objects: 2 objects, 449 KiB 2023-11-02T08:37:54.840 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: usage: 213 MiB used, 720 GiB / 720 GiB avail 2023-11-02T08:37:54.840 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: pgs: 42.285% pgs unknown 2023-11-02T08:37:54.840 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 55.713% pgs not active 2023-11-02T08:37:54.840 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 415 creating+peering 2023-11-02T08:37:54.840 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 359 unknown 2023-11-02T08:37:54.841 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 58 creating+activating 2023-11-02T08:37:54.841 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 17 active+clean 2023-11-02T08:37:54.841 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 2023-11-02T08:37:54.841 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 2023-11-02T08:37:54.841 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2925 ms) 2023-11-02T08:37:54.841 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [ RUN ] LibRadosService.Status 2023-11-02T08:37:54.842 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [ OK ] LibRadosService.Status (20066 ms) 2023-11-02T08:37:54.842 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [----------] 4 tests from LibRadosService (28639 ms total) 2023-11-02T08:37:54.842 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: 2023-11-02T08:37:54.842 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [----------] Global test environment tear-down 2023-11-02T08:37:54.842 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [==========] 4 tests from 1 test suite ran. (28639 ms total) 2023-11-02T08:37:54.843 INFO:tasks.workunit.client.0.smithi151.stdout: api_service: [ PASSED ] 4 tests. 2023-11-02T08:37:57.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-11-02T08:37:57.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [----------] Global test environment set-up. 2023-11-02T08:37:57.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-11-02T08:37:57.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-11-02T08:37:57.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5402 ms) 2023-11-02T08:37:57.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-11-02T08:37:57.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (125 ms) 2023-11-02T08:37:57.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-11-02T08:37:57.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20097 ms) 2023-11-02T08:37:57.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-11-02T08:37:57.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: attempt 0 of 20 2023-11-02T08:37:57.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5294 ms) 2023-11-02T08:37:57.134 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30918 ms total) 2023-11-02T08:37:57.134 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: 2023-11-02T08:37:57.134 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [----------] Global test environment tear-down 2023-11-02T08:37:57.134 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30918 ms total) 2023-11-02T08:37:57.134 INFO:tasks.workunit.client.0.smithi151.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-11-02T08:37:59.527 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:37:59.491+0000 12a62640 -1 osd.3 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 8.13 8.deba5533 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:37:59.581 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:02:40.472 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:59.648 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:40.524 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:59.658 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:40.533 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:59.659 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:02:40.544 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:59.663 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:37:59.618+0000 12a62640 -1 osd.4 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4.710d64fb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:37:59.672 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:02:40.562 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:59.743 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:02:40.633 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:59.789 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:40.665 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:59.875 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:02:40.765 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:37:59.905 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:02:40.796 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:00.617 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:38:00.470+0000 12a62640 -1 osd.3 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 8.13 8.deba5533 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:00.926 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:00.850+0000 12a62640 -1 osd.4 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4.710d64fb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:01.517 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:38:01.487+0000 12a62640 -1 osd.3 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 8.13 8.deba5533 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:01.559 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-11-02T08:38:01.559 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-11-02T08:38:01.560 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-11-02T08:38:01.560 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-11-02T08:38:01.560 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (35282 ms) 2023-11-02T08:38:01.560 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (35282 ms total) 2023-11-02T08:38:01.560 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: 2023-11-02T08:38:01.561 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-11-02T08:38:01.561 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (35282 ms total) 2023-11-02T08:38:01.561 INFO:tasks.workunit.client.0.smithi151.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-11-02T08:38:02.219 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:02.187+0000 12a62640 -1 osd.4 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4.710d64fb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:02.675 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:38:02.661+0000 12a62640 -1 osd.3 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 8.13 8:ccaa5d7b:test-rados-api-smithi151-91733-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:03.646 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:03.625+0000 12a62640 -1 osd.4 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4.710d64fb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:03.899 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:38:03.853+0000 12a62640 -1 osd.3 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 8.13 8:ccaa5d7b:test-rados-api-smithi151-91733-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:04.647 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:04.643+0000 12a62640 -1 osd.4 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4.710d64fb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:05.074 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:38:04.877+0000 12a62640 -1 osd.3 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 8.13 8:ccaa5d7b:test-rados-api-smithi151-91733-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:05.710 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:05.702+0000 12a62640 -1 osd.4 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4.710d64fb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:06.690 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:06.687+0000 12a62640 -1 osd.4 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4.710d64fb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:07.754 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:07.750+0000 12a62640 -1 osd.4 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4:df26b08e:test-rados-api-smithi151-91414-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:08.168 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:38:08.074+0000 12a62640 -1 osd.1 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 33.13 33.abd43df3 (undecoded) ondisk+write+known_if_redirected+full_try+supports_pool_eio e29) 2023-11-02T08:38:08.712 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:08.706+0000 12a62640 -1 osd.4 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4:df26b08e:test-rados-api-smithi151-91414-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:09.696 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:38:09.498+0000 12a62640 -1 osd.1 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 33.13 33.abd43df3 (undecoded) ondisk+write+known_if_redirected+full_try+supports_pool_eio e29) 2023-11-02T08:38:09.768 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:09.761+0000 12a62640 -1 osd.4 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4:df26b08e:test-rados-api-smithi151-91414-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:10.529 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:38:10.524+0000 12a62640 -1 osd.1 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4406.0:3 33.13 33:cfbc2bd5:::oid_PoolQuotaPP_0:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e29) 2023-11-02T08:38:10.875 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:38:10.846+0000 12a62640 -1 osd.4 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4371.0:2 4.1b 4:df26b08e:test-rados-api-smithi151-91414-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-11-02T08:38:21.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-11-02T08:38:21.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-11-02T08:38:21.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-11-02T08:38:21.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-11-02T08:38:21.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-11-02T08:38:21.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (616 ms) 2023-11-02T08:38:21.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-11-02T08:38:21.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (123 ms) 2023-11-02T08:38:21.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-11-02T08:38:21.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (54776 ms) 2023-11-02T08:38:21.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (55515 ms total) 2023-11-02T08:38:21.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: 2023-11-02T08:38:21.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-11-02T08:38:21.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (55516 ms total) 2023-11-02T08:38:21.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-11-02T08:38:36.102 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: Running main() from gmock_main.cc 2023-11-02T08:38:36.103 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-11-02T08:38:36.103 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [----------] Global test environment set-up. 2023-11-02T08:38:36.103 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-11-02T08:38:36.103 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-11-02T08:38:36.103 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (907 ms) 2023-11-02T08:38:36.104 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-11-02T08:38:36.104 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (312 ms) 2023-11-02T08:38:36.104 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-11-02T08:38:36.104 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (62638 ms) 2023-11-02T08:38:36.105 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-11-02T08:38:36.105 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:18.594276+0000 mon.b [INF] from='client.? 172.21.15.151:0/2560523728' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-91645-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-11-02T08:38:36.105 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:19.322551+0000 mon.a [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-11-02T08:38:36.105 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:19.325471+0000 mon.a [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 41 sec, daemons [osd.0,osd.4] have slow ops.) 2023-11-02T08:38:36.105 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:19.663517+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-91645-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-11-02T08:38:36.106 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:20.228505+0000 mon.a [WRN] pool 'PoolQuotaPP_smithi151-91372-3' is full (reached quota's max_bytes: 4 KiB) 2023-11-02T08:38:36.106 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:20.267362+0000 mon.a [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2023-11-02T08:38:36.106 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:21.809290+0000 mon.c [INF] from='client.? 172.21.15.151:0/2486589975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi151-91816-1"}]: dispatch 2023-11-02T08:38:36.106 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:21.840642+0000 mon.b [INF] from='client.? 172.21.15.151:0/446322058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-91630-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-11-02T08:38:36.107 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:23.494243+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-91630-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-11-02T08:38:36.107 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:23.608065+0000 mon.a [INF] from='client.? 172.21.15.151:0/3820154879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi151-91414-22"}]: dispatch 2023-11-02T08:38:36.107 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:24.346826+0000 mon.a [INF] from='client.? 172.21.15.151:0/1710182617' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi151-91464-10"}]: dispatch 2023-11-02T08:38:36.108 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:28.303421+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-91645-10","app": "rados","yes_i_really_mean_it": true}]': finished 2023-11-02T08:38:36.108 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:28.428813+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi151-91816-1"}]: dispatch 2023-11-02T08:38:36.108 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:28.824116+0000 mon.a [INF] from='client.? 172.21.15.151:0/1710182617' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi151-91464-10"}]: dispatch 2023-11-02T08:38:37.215 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:24.658132+0000 mon.b [INF] from='client.? 172.21.15.151:0/1458641706' entity='cl api_c_read_operations: Running main() from gmock_main.cc 2023-11-02T08:38:37.216 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-11-02T08:38:37.216 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-11-02T08:38:37.216 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-11-02T08:38:37.216 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-11-02T08:38:37.217 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-11-02T08:38:37.217 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-11-02T08:38:37.217 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (44642 ms) 2023-11-02T08:38:37.217 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-11-02T08:38:37.217 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (518 ms) 2023-11-02T08:38:37.218 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-11-02T08:38:37.218 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (363 ms) 2023-11-02T08:38:37.218 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-11-02T08:38:37.218 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (1163 ms) 2023-11-02T08:38:37.218 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-11-02T08:38:37.219 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (1637 ms) 2023-11-02T08:38:37.219 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-11-02T08:38:37.219 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (911 ms) 2023-11-02T08:38:37.219 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-11-02T08:38:37.219 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (221 ms) 2023-11-02T08:38:37.220 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-11-02T08:38:37.220 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (127 ms) 2023-11-02T08:38:37.220 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-11-02T08:38:37.220 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (207 ms) 2023-11-02T08:38:37.220 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-11-02T08:38:37.221 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (96 ms) 2023-11-02T08:38:37.221 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-11-02T08:38:37.221 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (471 ms) 2023-11-02T08:38:37.221 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-11-02T08:38:37.221 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (2527 ms) 2023-11-02T08:38:37.221 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-11-02T08:38:37.222 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (859 ms) 2023-11-02T08:38:37.222 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-11-02T08:38:37.222 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (270 ms) 2023-11-02T08:38:37.222 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-11-02T08:38:37.222 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (222 ms) 2023-11-02T08:38:37.223 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-11-02T08:38:37.223 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (124 ms) 2023-11-02T08:38:37.223 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (54358 ms total) 2023-11-02T08:38:37.223 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: 2023-11-02T08:38:37.223 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-11-02T08:38:37.224 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (71002 ms total) 2023-11-02T08:38:37.224 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-11-02T08:38:42.187 INFO:tasks.workunit.client.0.smithi151.stdout:ient.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi151-91515-10"}]: dispatch 2023-11-02T08:38:42.187 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:29.276712+0000 mon.a [INF] from='client.? 172.21.15.151:0/3820154879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi151-91414-22"}]: dispatch 2023-11-02T08:38:42.188 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:29.488697+0000 mon.a [INF] from='client.? 172.21.15.151:0/1710182617' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi151-91464-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-11-02T08:38:42.188 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:29.782835+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi151-91515-10"}]: dispatch 2023-11-02T08:38:42.188 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:31.057314+0000 mon.a [WRN] Health check failed: Slow OSD heartbeats on back (longest 18368.300ms) (OSD_SLOW_PING_TIME_BACK) 2023-11-02T08:38:42.188 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:31.058219+0000 mon.a [WRN] Health check failed: Slow OSD heartbeats on front (longest 18404.459ms) (OSD_SLOW_PING_TIME_FRONT) 2023-11-02T08:38:42.188 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:31.058903+0000 mon.a [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-11-02T08:38:42.189 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:31.917677+0000 mon.a [INF] from='client.? 172.21.15.151:0/2311267875' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi151-91774-7"}]: dispatch 2023-11-02T08:38:42.189 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:31.975225+0000 client.admin [INF] onexx 2023-11-02T08:38:42.189 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:31.988887+0000 mon.a [INF] from='client.? 172.21.15.151:0/3665579392' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-11-02T08:38:42.189 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:32.195143+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi151-91630-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-11-02T08:38:42.190 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:32.198650+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi151-91816-1"}]': finished 2023-11-02T08:38:42.190 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:32.555919+0000 mon.a [INF] from='client.? 172.21.15.151:0/2311267875' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi151-91774-7"}]: dispatch 2023-11-02T08:38:42.190 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:32.585533+0000 mon.b [INF] from='client.? 172.21.15.151:0/3936192564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi151-91736-7"}]: dispatch 2023-11-02T08:38:42.190 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:33.087421+0000 mon.a [INF] from='client.? 172.21.15.151:0/2311267875' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi151-91774-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-11-02T08:38:42.191 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:33.187150+0000 mon.a [INF] from='client.? 172.21.15.151:0/3820154879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi151-91414-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-11-02T08:38:42.191 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:33.293681+0000 mon.b [INF] from='client.? 172.21.15.151:0/897552540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi151-91394-16"}]: dispatch 2023-11-02T08:38:42.191 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:33.329717+0000 mon.c [INF] from='client.? 172.21.15.151:0/2486589975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi151-91816-1"}]: dispatch 2023-11-02T08:38:43.435 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:33.617539+0000 mon.b [INF] from='client.? 172.21.15.151:0/1458641706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi151-91515-10"}]: dispatch 2023-11-02T08:38:43.435 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:33.857511+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi151-91736-7"}]: dispatch 2023-11-02T08:38:43.435 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:34.209256+0000 mon.c [INF] from='client.11536 172.21.15.151:0/897552540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi151-91394-16"}]: dispatch 2023-11-02T08:38:43.436 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:35.300876+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi151-91816-1"}]: dispatch 2023-11-02T08:38:43.436 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:35.390002+0000 mon.a [INF] from='client.11536 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi151-91394-16"}]: dispatch 2023-11-02T08:38:43.436 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:35.440501+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi151-91515-10"}]: dispatch 2023-11-02T08:38:43.436 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:35.608809+0000 mon.a [INF] from='client.? 172.21.15.151:0/1710182617' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi151-91464-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-11-02T08:38:43.436 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:35.725997+0000 mon.a [INF] from='client.? 172.21.15.151:0/3665579392' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2023-11-02T08:38:43.437 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:36.213206+0000 client.admin [INF] twoxx 2023-11-02T08:38:43.437 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:36.225553+0000 mon.a [INF] from='client.? 172.21.15.151:0/3665579392' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-11-02T08:38:43.437 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:36.272541+0000 mon.a [INF] from='client.? 172.21.15.151:0/1710182617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi151-91464-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi151-91464-10"}]: dispatch 2023-11-02T08:38:43.437 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:36.915523+0000 mon.b [INF] from='client.? 172.21.15.151:0/1458641706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi151-91515-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-11-02T08:38:43.437 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:37.166976+0000 mon.b [INF] from='client.? 172.21.15.151:0/3936192564' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi151-91736-7"}]: dispatch 2023-11-02T08:38:43.438 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:37.686574+0000 mon.c [INF] from='client.11536 172.21.15.151:0/897552540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi151-91394-16"}]: dispatch 2023-11-02T08:38:43.438 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:37.956534+0000 mon.c [INF] from='client.? 172.21.15.151:0/3898478209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi151-91355-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-11-02T08:38:43.438 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:38.582050+0000 mon.a [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-11-02T08:38:43.438 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:38.876442+0000 mon.a [INF] from='client.11536 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi151-91394-16"}]: dispatch 2023-11-02T08:38:44.927 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:03:25.817 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:44.943 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:03:25.833 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:45.098 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:03:25.989 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:45.543 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:03:26.433 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:45.574 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:26.450 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:45.586 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:26.462 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:45.656 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:03:26.547 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:45.684 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:26.559 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:45.862 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:03:26.753 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:38:47.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:38.895816+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi151-91355-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-11-02T08:38:47.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:38.971525+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi151-91515-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-11-02T08:38:47.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:38.982315+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi151-91736-7"}]: dispatch 2023-11-02T08:38:47.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:39.273328+0000 mon.a [INF] from='client.? 172.21.15.151:0/2311267875' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi151-91774-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-11-02T08:38:47.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:39.275099+0000 mon.a [INF] from='client.? 172.21.15.151:0/3820154879' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi151-91414-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-11-02T08:38:47.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:39.277334+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi151-91816-1"}]': finished 2023-11-02T08:38:47.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:39.407593+0000 mon.a [INF] from='client.? 172.21.15.151:0/3665579392' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-11-02T08:38:47.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:39.718771+0000 mon.a [INF] from='client.? 172.21.15.151:0/2311267875' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi151-91774-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi151-91774-7"}]: dispatch 2023-11-02T08:38:47.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:39.296699+0000 mon.c [INF] from='client.? 172.21.15.151:0/1696455668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi151-91843-12"}]: dispatch 2023-11-02T08:38:47.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:40.501352+0000 mon.c [INF] from='client.11536 172.21.15.151:0/897552540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi151-91394-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-11-02T08:38:47.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:40.513021+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi151-91843-12"}]: dispatch 2023-11-02T08:38:47.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:40.516968+0000 mon.a [INF] from='client.? 172.21.15.151:0/3820154879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi151-91414-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi151-91414-22"}]: dispatch 2023-11-02T08:38:47.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:40.768513+0000 mon.b [INF] from='client.? 172.21.15.151:0/2560523728' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi151-91645-10", "tierpool":"test-rados-api-smithi151-91645-10-cache", "force_nonempty":""}]: dispatch 2023-11-02T08:38:47.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:40.898260+0000 mon.b [INF] from='client.? 172.21.15.151:0/3936192564' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi151-91736-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-11-02T08:38:47.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:41.921394+0000 client.admin [INF] threexx 2023-11-02T08:38:47.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:41.959117+0000 mon.a [INF] from='client.? 172.21.15.151:0/3665579392' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-11-02T08:38:48.620 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2 api_list: [==========] Running 11 tests from 3 test suites. 2023-11-02T08:38:48.621 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [----------] Global test environment set-up. 2023-11-02T08:38:48.621 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [----------] 7 tests from LibRadosList 2023-11-02T08:38:48.621 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-11-02T08:38:48.621 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosList.ListObjects (47460 ms) 2023-11-02T08:38:48.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-11-02T08:38:48.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (2631 ms) 2023-11-02T08:38:48.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-11-02T08:38:48.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: myset foo1,foo2,foo3 2023-11-02T08:38:48.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo1 2023-11-02T08:38:48.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo2 2023-11-02T08:38:48.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo3 2023-11-02T08:38:48.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: myset foo1,foo4,foo5 2023-11-02T08:38:48.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo4 2023-11-02T08:38:48.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo5 2023-11-02T08:38:48.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo1 2023-11-02T08:38:48.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: myset foo6,foo7 2023-11-02T08:38:48.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo7 2023-11-02T08:38:48.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo6 2023-11-02T08:38:48.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-11-02T08:38:48.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns1:foo4 2023-11-02T08:38:48.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns1:foo5 2023-11-02T08:38:48.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns2:foo7 2023-11-02T08:38:48.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns2:foo6 2023-11-02T08:38:48.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns1:foo1 2023-11-02T08:38:48.626 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: :foo1 2023-11-02T08:38:48.626 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: :foo2 2023-11-02T08:38:48.626 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: :foo3 2023-11-02T08:38:48.627 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (22022 ms) 2023-11-02T08:38:48.627 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-11-02T08:38:48.627 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 1 0 2023-11-02T08:38:48.627 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 10 0 2023-11-02T08:38:48.628 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 13 0 2023-11-02T08:38:48.628 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 7 0 2023-11-02T08:38:48.628 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 14 0 2023-11-02T08:38:48.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 0 0 2023-11-02T08:38:48.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 15 0 2023-11-02T08:38:48.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 11 0 2023-11-02T08:38:48.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 5 0 2023-11-02T08:38:48.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 8 0 2023-11-02T08:38:48.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 6 0 2023-11-02T08:38:48.630 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 3 0 2023-11-02T08:38:48.630 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 4 0 2023-11-02T08:38:48.630 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 12 0 2023-11-02T08:38:48.631 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 9 0 2023-11-02T08:38:48.631 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 2 0 2023-11-02T08:38:48.631 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-11-02T08:38:48.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (2004 ms) 2023-11-02T08:38:48.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-11-02T08:38:48.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: x cursor=MIN 2023-11-02T08:38:48.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=1 cursor=9:02547ec2:::1:head 2023-11-02T08:38:48.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=10 cursor=9:52ea6a34:::10:head 2023-11-02T08:38:48.633 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=13 cursor=9:566253c9:::13:head 2023-11-02T08:38:48.633 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=7 cursor=9:5c6b0b28:::7:head 2023-11-02T08:38:48.633 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=14 cursor=9:62a1935d:::14:head 2023-11-02T08:38:48.633 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=0 cursor=9:6cac518f:::0:head 2023-11-02T08:38:48.633 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=15 cursor=9:863748b0:::15:head 2023-11-02T08:38:48.634 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=11 cursor=9:89d3ae78:::11:head 2023-11-02T08:38:48.634 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=5 cursor=9:b29083e3:::5:head 2023-11-02T08:38:48.634 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=8 cursor=9:bd63b0f1:::8:head 2023-11-02T08:38:48.634 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=6 cursor=9:c4fdafeb:::6:head 2023-11-02T08:38:48.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=3 cursor=9:cfc208b3:::3:head 2023-11-02T08:38:48.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=4 cursor=9:d83876eb:::4:head 2023-11-02T08:38:48.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=12 cursor=9:de5d7c5f:::12:head 2023-11-02T08:38:48.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=9 cursor=9:e960b815:::9:head 2023-11-02T08:38:48.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > oid=2 cursor=9:f905c69b:::2:head 2023-11-02T08:38:48.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: FIRST> seek to MIN oid=1 2023-11-02T08:38:48.636 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=1 cursor=9:02547ec2:::1:head 2023-11-02T08:38:48.636 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:02547ec2:::1:head 2023-11-02T08:38:48.636 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:02547ec2:::1:head -> 1 2023-11-02T08:38:48.636 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=10 cursor=9:52ea6a34:::10:head 2023-11-02T08:38:48.637 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:52ea6a34:::10:head 2023-11-02T08:38:48.637 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2023-11-02T08:38:48.637 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=13 cursor=9:566253c9:::13:head 2023-11-02T08:38:48.637 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:566253c9:::13:head 2023-11-02T08:38:48.637 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:566253c9:::13:head -> 13 2023-11-02T08:38:48.638 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=7 cursor=9:5c6b0b28:::7:head 2023-11-02T08:38:48.638 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2023-11-02T08:38:48.638 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2023-11-02T08:38:50.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=14 cursor=9:62a1935d:::14:head 2023-11-02T08:38:50.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:62a1935d:::14:head 2023-11-02T08:38:50.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:62a1935d:::14:head -> 14 2023-11-02T08:38:50.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=0 cursor=9:6cac518f:::0:head 2023-11-02T08:38:50.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:6cac518f:::0:head 2023-11-02T08:38:50.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:6cac518f:::0:head -> 0 2023-11-02T08:38:50.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=15 cursor=9:863748b0:::15:head 2023-11-02T08:38:50.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:863748b0:::15:head 2023-11-02T08:38:50.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:863748b0:::15:head -> 15 2023-11-02T08:38:50.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=11 cursor=9:89d3ae78:::11:head 2023-11-02T08:38:50.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:89d3ae78:::11:head 2023-11-02T08:38:50.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2023-11-02T08:38:50.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=5 cursor=9:b29083e3:::5:head 2023-11-02T08:38:50.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:b29083e3:::5:head 2023-11-02T08:38:50.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:b29083e3:::5:head -> 5 2023-11-02T08:38:50.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=8 cursor=9:bd63b0f1:::8:head 2023-11-02T08:38:50.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2023-11-02T08:38:50.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2023-11-02T08:38:50.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=6 cursor=9:c4fdafeb:::6:head 2023-11-02T08:38:50.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2023-11-02T08:38:50.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2023-11-02T08:38:50.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=3 cursor=9:cfc208b3:::3:head 2023-11-02T08:38:50.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:cfc208b3:::3:head 2023-11-02T08:38:50.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2023-11-02T08:38:50.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=4 cursor=9:d83876eb:::4:head 2023-11-02T08:38:50.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:d83876eb:::4:head 2023-11-02T08:38:50.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:d83876eb:::4:head -> 4 2023-11-02T08:38:50.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=12 cursor=9:de5d7c5f:::12:head 2023-11-02T08:38:50.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2023-11-02T08:38:50.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2023-11-02T08:38:50.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=9 cursor=9:e960b815:::9:head 2023-11-02T08:38:50.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:e960b815:::9:head 2023-11-02T08:38:50.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:e960b815:::9:head -> 9 2023-11-02T08:38:50.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : oid=2 cursor=9:f905c69b:::2:head 2023-11-02T08:38:50.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:f905c69b:::2:head 2023-11-02T08:38:50.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:f905c69b:::2:head -> 2 2023-11-02T08:38:50.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:6cac518f:::0:head 2023-11-02T08:38:50.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:6cac518f:::0:head expected=9:6cac518f:::0:head 2023-11-02T08:38:50.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:6cac518f:::0:head -> 0 2023-11-02T08:38:50.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=0 expected=0 2023-11-02T08:38:50.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:52ea6a34:::10:head 2023-11-02T08:38:50.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:52ea6a34:::10:head expected=9:52ea6a34:::10:head 2023-11-02T08:38:50.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2023-11-02T08:38:50.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=10 expected=10 2023-11-02T08:38:50.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:566253c9:::13:head 2023-11-02T08:38:50.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:566253c9:::13:head expected=9:566253c9:::13:head 2023-11-02T08:38:50.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:566253c9:::13:head -> 13 2023-11-02T08:38:50.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=13 expected=13 2023-11-02T08:38:50.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:02547ec2:::1:head 2023-11-02T08:38:50.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:02547ec2:::1:head expected=9:02547ec2:::1:head 2023-11-02T08:38:50.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:02547ec2:::1:head -> 1 2023-11-02T08:38:50.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=1 expected=1 2023-11-02T08:38:50.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:e960b815:::9:head 2023-11-02T08:38:50.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:e960b815:::9:head expected=9:e960b815:::9:head 2023-11-02T08:38:50.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:e960b815:::9:head -> 9 2023-11-02T08:38:50.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=9 expected=9 2023-11-02T08:38:50.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2023-11-02T08:38:50.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:de5d7c5f:::12:head expected=9:de5d7c5f:::12:head 2023-11-02T08:38:50.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2023-11-02T08:38:50.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=12 expected=12 2023-11-02T08:38:50.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:d83876eb:::4:head 2023-11-02T08:38:50.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:d83876eb:::4:head expected=9:d83876eb:::4:head 2023-11-02T08:38:50.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:d83876eb:::4:head -> 4 2023-11-02T08:38:50.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=4 expected=4 2023-11-02T08:38:50.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2023-11-02T08:38:50.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:5c6b0b28:::7:head expected=9:5c6b0b28:::7:head 2023-11-02T08:38:50.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2023-11-02T08:38:50.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=7 expected=7 2023-11-02T08:38:50.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:f905c69b:::2:head 2023-11-02T08:38:53.404 INFO:tasks.workunit.client.0.smithi151.stdout: 023-11-02T08:38:42.273438+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi151-91645-10", "tierpool":"test-rados-api-smithi151-91645-10-cache", "force_nonempty":""}]: dispatch 2023-11-02T08:38:53.404 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:42.289496+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi151-91736-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-11-02T08:38:53.404 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:42.941706+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi151-91355-5","app": "rados","yes_i_really_mean_it": true}]': finished 2023-11-02T08:38:53.404 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:42.952518+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi151-91515-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-11-02T08:38:53.405 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: got: 2023-11-02T08:38:43.142026+0000 mon.a [INF] from='client.11536 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi151-91394-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-11-02T08:38:53.405 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (23416 ms) 2023-11-02T08:38:53.405 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (87274 ms total) 2023-11-02T08:38:53.405 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: 2023-11-02T08:38:53.405 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [----------] Global test environment tear-down 2023-11-02T08:38:53.406 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (87274 ms total) 2023-11-02T08:38:53.406 INFO:tasks.workunit.client.0.smithi151.stdout: api_cmd: [ PASSED ] 4 tests. 2023-11-02T08:39:09.263 INFO:tasks.ceph.mon.c.smithi151.stderr:2023-11-02T08:39:09.231+0000 fb3f640 -1 mon.c@2(peon).paxos(paxos updating c 1..278) lease_expire from mon.0 v2:172.21.15.151:3300/0 is 0.455352992s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2023-11-02T08:39:17.809 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_1_[92177]: starting. 2023-11-02T08:39:17.809 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_1_[92177]: creating pool ceph_test_rados_open_pools_parallel.smithi151-92139 2023-11-02T08:39:17.810 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_1_[92177]: created object 0... 2023-11-02T08:39:17.810 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_1_[92177]: created object 25... 2023-11-02T08:39:17.810 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_1_[92177]: created object 49... 2023-11-02T08:39:17.811 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_1_[92177]: finishing. 2023-11-02T08:39:17.811 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_1_[92177]: shutting down. 2023-11-02T08:39:17.811 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_2_[92181]: starting. 2023-11-02T08:39:17.812 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_2_[92181]: rados_pool_create. 2023-11-02T08:39:17.812 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_2_[92181]: rados_ioctx_create. 2023-11-02T08:39:17.812 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_2_[92181]: shutting down. 2023-11-02T08:39:17.812 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: ******************************* 2023-11-02T08:39:17.812 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_3_[118444]: starting. 2023-11-02T08:39:17.813 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_3_[118444]: creating pool ceph_test_rados_open_pools_parallel.smithi151-92139 2023-11-02T08:39:17.813 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_3_[118444]: created object 0... 2023-11-02T08:39:17.813 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_3_[118444]: created object 25... 2023-11-02T08:39:17.813 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_3_[118444]: created object 49... 2023-11-02T08:39:17.813 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_3_[118444]: finishing. 2023-11-02T08:39:17.814 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_3_[118444]: shutting down. 2023-11-02T08:39:17.814 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: ******************************* 2023-11-02T08:39:17.814 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_4_[118445]: starting. 2023-11-02T08:39:17.814 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_4_[118445]: rados_pool_create. 2023-11-02T08:39:17.815 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_4_[118445]: rados_ioctx_create. 2023-11-02T08:39:17.815 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: process_4_[118445]: shutting down. 2023-11-02T08:39:17.815 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: ******************************* 2023-11-02T08:39:17.816 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: ******************************* 2023-11-02T08:39:17.816 INFO:tasks.workunit.client.0.smithi151.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-11-02T08:39:17.826 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2023-11-02T08:39:17.826 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] Global test environment set-up. 2023-11-02T08:39:17.827 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-11-02T08:39:17.827 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-11-02T08:39:17.827 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-11-02T08:39:17.828 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-11-02T08:39:17.828 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: 2023-11-02T08:39:17.829 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-11-02T08:39:17.829 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-11-02T08:39:17.830 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2023-11-02T08:37:26.236+0000 7fab13cf9780 -1 failed for service _ceph-mon._tcp 2023-11-02T08:39:17.830 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: ceph-mon 2023-11-02T08:39:17.830 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: 2023-11-02T08:37:26.236+0000 7fab13cf9780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-11-02T08:39:17.831 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (243 ms) 2023-11-02T08:39:17.831 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-11-02T08:39:17.831 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2014 ms) 2023-11-02T08:39:17.832 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2257 ms total) 2023-11-02T08:39:17.832 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: 2023-11-02T08:39:17.832 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-11-02T08:39:17.833 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-11-02T08:39:17.833 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: started 0x5648565d22d0 2023-11-02T08:39:17.834 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: started 0x564856fd4e30 2023-11-02T08:39:17.834 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: started 0x5648565d3590 2023-11-02T08:39:17.834 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: started 0x56485674be50 2023-11-02T08:39:17.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: started 0x5648566a3ce0 2023-11-02T08:39:17.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: started 0x56485674cde0 2023-11-02T08:39:17.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: started 6 aios 2023-11-02T08:39:17.835 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: waiting 0x5648565d22d0 2023-11-02T08:39:17.836 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: waiting 0x564856fd4e30 2023-11-02T08:39:17.836 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: waiting 0x5648565d3590 2023-11-02T08:39:17.836 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: waiting 0x56485674be50 2023-11-02T08:39:17.836 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: waiting 0x5648566a3ce0 2023-11-02T08:39:17.837 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: waiting 0x56485674cde0 2023-11-02T08:39:17.837 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: done. 2023-11-02T08:39:17.837 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (12388 ms) 2023-11-02T08:39:17.838 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (12388 ms total) 2023-11-02T08:39:17.838 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: 2023-11-02T08:39:17.838 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-11-02T08:39:17.838 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-11-02T08:39:17.839 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-11-02T08:39:17.840 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-11-02T08:39:17.840 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMisc.Exec (12630 ms) 2023-11-02T08:39:17.840 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-11-02T08:39:17.841 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (2412 ms) 2023-11-02T08:39:17.841 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-11-02T08:39:17.841 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (847 ms) 2023-11-02T08:39:17.842 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-11-02T08:39:17.842 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5866 ms) 2023-11-02T08:39:17.842 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-11-02T08:39:17.842 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-11-02T08:39:17.843 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-11-02T08:39:17.843 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-11-02T08:39:17.843 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-11-02T08:39:17.844 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (71894 ms) 2023-11-02T08:39:17.844 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] 8 tests from LibRadosMisc (93649 ms total) 2023-11-02T08:39:17.844 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: 2023-11-02T08:39:17.844 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [----------] Global test environment tear-down 2023-11-02T08:39:17.845 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (111816 ms total) 2023-11-02T08:39:17.845 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc: [ PASSED ] 12 tests. 2023-11-02T08:39:18.007 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: Running main() from gmock_main.cc 2023-11-02T08:39:18.007 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-11-02T08:39:18.007 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [----------] Global test environment set-up. 2023-11-02T08:39:18.007 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-11-02T08:39:18.008 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: seed 91414 2023-11-02T08:39:18.008 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-11-02T08:39:18.008 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-11-02T08:39:18.008 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-11-02T08:39:18.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (43250 ms) 2023-11-02T08:39:18.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-11-02T08:39:18.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (173 ms) 2023-11-02T08:39:18.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-11-02T08:39:18.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (80 ms) 2023-11-02T08:39:18.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-11-02T08:39:18.010 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (39 ms) 2023-11-02T08:39:18.010 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-11-02T08:39:18.010 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (205 ms) 2023-11-02T08:39:18.010 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-11-02T08:39:18.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (88 ms) 2023-11-02T08:39:18.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-11-02T08:39:18.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (47 ms) 2023-11-02T08:39:18.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-11-02T08:39:18.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (210 ms) 2023-11-02T08:39:18.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-11-02T08:39:18.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (96 ms) 2023-11-02T08:39:18.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-11-02T08:39:18.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (66 ms) 2023-11-02T08:39:18.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-11-02T08:39:18.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (227 ms) 2023-11-02T08:39:18.013 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-11-02T08:39:18.013 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (98 ms) 2023-11-02T08:39:18.013 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-11-02T08:39:18.013 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (79 ms) 2023-11-02T08:39:18.013 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-11-02T08:39:18.014 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (106 ms) 2023-11-02T08:39:18.014 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-11-02T08:39:18.014 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (1588 ms) 2023-11-02T08:39:18.014 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-11-02T08:39:18.014 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (377 ms) 2023-11-02T08:39:18.015 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-11-02T08:39:18.015 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (544 ms) 2023-11-02T08:39:18.015 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-11-02T08:39:18.015 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (89 ms) 2023-11-02T08:39:18.015 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-11-02T08:39:18.015 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (120 ms) 2023-11-02T08:39:18.016 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (47483 ms total) 2023-11-02T08:39:18.016 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: 2023-11-02T08:39:18.016 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-11-02T08:39:18.016 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-11-02T08:39:18.016 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (7177 ms) 2023-11-02T08:39:18.017 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-11-02T08:39:18.017 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (14951 ms) 2023-11-02T08:39:18.017 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-11-02T08:39:18.017 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (241 ms) 2023-11-02T08:39:18.017 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-11-02T08:39:18.017 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (655 ms) 2023-11-02T08:39:18.018 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-11-02T08:39:18.018 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (692 ms) 2023-11-02T08:39:18.896 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-11-02T08:39:18.896 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-11-02T08:39:18.897 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-11-02T08:39:18.897 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-11-02T08:39:18.897 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: seed 91774 2023-11-02T08:39:18.898 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-11-02T08:39:18.898 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (16842 ms) 2023-11-02T08:39:18.898 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-11-02T08:39:18.899 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (25141 ms) 2023-11-02T08:39:18.899 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-11-02T08:39:18.899 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (2187 ms) 2023-11-02T08:39:18.900 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-11-02T08:39:18.900 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (1956 ms) 2023-11-02T08:39:18.900 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-11-02T08:39:18.900 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (498 ms) 2023-11-02T08:39:18.901 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (46624 ms total) 2023-11-02T08:39:18.901 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: 2023-11-02T08:39:18.902 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-11-02T08:39:18.902 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-11-02T08:39:18.903 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (13021 ms) 2023-11-02T08:39:18.903 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-11-02T08:39:18.904 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (577 ms) 2023-11-02T08:39:18.905 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-11-02T08:39:18.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1778 ms) 2023-11-02T08:39:18.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-11-02T08:39:18.906 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (2015 ms) 2023-11-02T08:39:18.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (17391 ms total) 2023-11-02T08:39:18.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: 2023-11-02T08:39:18.907 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-11-02T08:39:18.908 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (112791 ms total) 2023-11-02T08:39:18.908 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-11-02T08:39:18.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: Running main() from gmock_main.cc 2023-11-02T08:39:18.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-11-02T08:39:18.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [----------] Global test environment set-up. 2023-11-02T08:39:18.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-11-02T08:39:18.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-11-02T08:39:18.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ OK ] LibRadosPools.PoolList (5730 ms) 2023-11-02T08:39:18.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-11-02T08:39:18.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4167 ms) 2023-11-02T08:39:18.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-11-02T08:39:18.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (5691 ms) 2023-11-02T08:39:18.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-11-02T08:39:18.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3792 ms) 2023-11-02T08:39:18.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-11-02T08:39:18.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3514 ms) 2023-11-02T08:39:18.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-11-02T08:39:18.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5625 ms) 2023-11-02T08:39:18.937 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-11-02T08:39:18.937 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5910 ms) 2023-11-02T08:39:18.938 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-11-02T08:39:18.938 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (7222 ms) 2023-11-02T08:39:18.938 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-11-02T08:39:18.939 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (71232 ms) 2023-11-02T08:39:18.939 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [----------] 9 tests from LibRadosPools (112883 ms total) 2023-11-02T08:39:18.939 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: 2023-11-02T08:39:18.940 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [----------] Global test environment tear-down 2023-11-02T08:39:18.940 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (112883 ms total) 2023-11-02T08:39:18.940 INFO:tasks.workunit.client.0.smithi151.stdout: api_pool: [ PASSED ] 9 tests. 2023-11-02T08:39:21.188 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: Running main() from gmock_main.cc 2023-11-02T08:39:21.188 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-11-02T08:39:21.189 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [----------] Global test environment set-up. 2023-11-02T08:39:21.189 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-11-02T08:39:21.189 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-11-02T08:39:21.189 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ OK ] LibRadosStat.Stat (22432 ms) 2023-11-02T08:39:21.189 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-11-02T08:39:21.190 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (12446 ms) 2023-11-02T08:39:21.190 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-11-02T08:39:21.190 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ OK ] LibRadosStat.StatNS (7386 ms) 2023-11-02T08:39:21.191 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-11-02T08:39:21.191 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (2454 ms) 2023-11-02T08:39:21.191 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-11-02T08:39:21.191 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (1420 ms) 2023-11-02T08:39:21.191 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [----------] 5 tests from LibRadosStat (46138 ms total) 2023-11-02T08:39:21.192 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: 2023-11-02T08:39:21.192 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-11-02T08:39:21.192 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-11-02T08:39:21.192 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (14797 ms) 2023-11-02T08:39:21.192 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-11-02T08:39:21.193 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (1258 ms) 2023-11-02T08:39:21.193 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-11-02T08:39:21.193 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (3 ms) 2023-11-02T08:39:21.193 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-11-02T08:39:21.193 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (112 ms) 2023-11-02T08:39:21.194 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (16170 ms total) 2023-11-02T08:39:21.194 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: 2023-11-02T08:39:21.194 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [----------] Global test environment tear-down 2023-11-02T08:39:21.195 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (115101 ms total) 2023-11-02T08:39:21.195 INFO:tasks.workunit.client.0.smithi151.stdout: api_stat: [ PASSED ] 9 tests. 2023-11-02T08:39:21.614 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:04:02.505 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:21.619 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:04:02.509 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:21.633 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:02.508 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:21.640 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:02.516 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:21.643 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:04:02.533 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:21.709 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:04:02.599 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:21.818 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:04:02.708 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:21.837 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:04:02.728 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:21.854 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:02.729 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:28.966 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-11-02T08:39:28.966 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-11-02T08:39:28.966 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-11-02T08:39:28.967 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-11-02T08:39:28.967 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-11-02T08:39:28.967 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify_test_cb 2023-11-02T08:39:28.967 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (41941 ms) 2023-11-02T08:39:28.967 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-11-02T08:39:28.968 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94347207009872 err -107 2023-11-02T08:39:28.968 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-11-02T08:39:28.968 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (604 ms) 2023-11-02T08:39:28.968 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-11-02T08:39:28.968 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-11-02T08:39:28.969 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94347207021744 err -107 2023-11-02T08:39:28.969 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1973 ms) 2023-11-02T08:39:28.969 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-11-02T08:39:28.969 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_cb from 4492 notify_id 240518168576 cookie 94347207021744 2023-11-02T08:39:28.969 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (362 ms) 2023-11-02T08:39:28.970 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-11-02T08:39:28.970 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_cb from 4492 notify_id 240518168576 cookie 94347207038592 2023-11-02T08:39:28.970 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (574 ms) 2023-11-02T08:39:28.970 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-11-02T08:39:28.970 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_cb from 4492 notify_id 240518168576 cookie 94347207038592 2023-11-02T08:39:28.971 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (356 ms) 2023-11-02T08:39:28.971 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-11-02T08:39:28.971 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_cb from 4492 notify_id 240518168576 cookie 94347207075632 2023-11-02T08:39:28.971 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_cb from 4492 notify_id 240518168576 cookie 94347207088528 2023-11-02T08:39:28.971 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (381 ms) 2023-11-02T08:39:28.972 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-11-02T08:39:28.972 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_cb from 4492 notify_id 240518168577 cookie 94347207088528 2023-11-02T08:39:28.972 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_cb from 4492 notify_id 240518168578 cookie 94347207088528 2023-11-02T08:39:28.972 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3303 ms) 2023-11-02T08:39:28.972 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-11-02T08:39:28.973 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-11-02T08:39:28.973 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94347207088528 err -107 2023-11-02T08:39:28.973 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_cb from 4492 notify_id 249108103170 cookie 94347207088496 2023-11-02T08:39:28.973 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5431 ms) 2023-11-02T08:39:28.973 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-11-02T08:39:28.974 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94347207075632 err -107 2023-11-02T08:39:28.974 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-11-02T08:39:28.974 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (100 ms) 2023-11-02T08:39:28.974 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (55025 ms total) 2023-11-02T08:39:28.975 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: 2023-11-02T08:39:28.975 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-11-02T08:39:28.975 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-11-02T08:39:28.975 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: watch_notify_test_cb 2023-11-02T08:39:28.975 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (9075 ms) 2023-11-02T08:39:28.976 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (9075 ms total) 2023-11-02T08:39:28.976 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: 2023-11-02T08:39:28.976 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-11-02T08:39:28.976 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (122833 ms total) 2023-11-02T08:39:28.977 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-11-02T08:39:34.918 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-11-02T08:39:34.918 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-11-02T08:39:34.919 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-11-02T08:39:34.919 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-11-02T08:39:34.919 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: seed 91515 2023-11-02T08:39:34.920 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-11-02T08:39:34.920 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (15112 ms) 2023-11-02T08:39:34.920 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-11-02T08:39:34.920 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (6823 ms) 2023-11-02T08:39:34.921 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-11-02T08:39:34.921 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (7786 ms) 2023-11-02T08:39:34.921 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-11-02T08:39:34.921 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (4519 ms) 2023-11-02T08:39:34.921 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-11-02T08:39:34.922 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (2798 ms) 2023-11-02T08:39:34.922 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-11-02T08:39:34.922 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (2475 ms) 2023-11-02T08:39:34.922 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-11-02T08:39:34.922 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4334 ms) 2023-11-02T08:39:34.923 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-11-02T08:39:34.923 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (480 ms) 2023-11-02T08:39:34.923 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (44327 ms total) 2023-11-02T08:39:34.923 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: 2023-11-02T08:39:34.923 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-11-02T08:39:34.924 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-11-02T08:39:34.924 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (17397 ms) 2023-11-02T08:39:34.924 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-11-02T08:39:34.924 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (163 ms) 2023-11-02T08:39:34.924 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-11-02T08:39:34.925 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1494 ms) 2023-11-02T08:39:34.925 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-11-02T08:39:34.925 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (3426 ms) 2023-11-02T08:39:34.925 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-11-02T08:39:34.926 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (3864 ms) 2023-11-02T08:39:34.926 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-11-02T08:39:34.926 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (3086 ms) 2023-11-02T08:39:34.926 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-11-02T08:39:34.926 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (1096 ms) 2023-11-02T08:39:34.927 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-11-02T08:39:34.927 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3046 ms) 2023-11-02T08:39:34.927 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (33572 ms total) 2023-11-02T08:39:34.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: 2023-11-02T08:39:34.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-11-02T08:39:34.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (128925 ms total) 2023-11-02T08:39:34.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-11-02T08:39:34.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-11-02T08:39:34.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-11-02T08:39:34.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-11-02T08:39:34.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-11-02T08:39:34.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-11-02T08:39:34.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-11-02T08:39:34.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-11-02T08:39:34.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: 2023-11-02T08:39:34.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-11-02T08:39:34.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-11-02T08:39:34.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (11503 ms) 2023-11-02T08:39:34.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-11-02T08:39:34.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (14580 ms) 2023-11-02T08:39:34.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-11-02T08:39:34.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (41817 ms) 2023-11-02T08:39:34.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-11-02T08:39:34.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (29990 ms) 2023-11-02T08:39:34.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-11-02T08:39:34.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (20958 ms) 2023-11-02T08:39:34.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-11-02T08:39:34.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (4933 ms) 2023-11-02T08:39:34.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-11-02T08:39:34.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (4925 ms) 2023-11-02T08:39:34.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (128707 ms total) 2023-11-02T08:39:34.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: 2023-11-02T08:39:34.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-11-02T08:39:34.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (128707 ms total) 2023-11-02T08:39:34.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-11-02T08:39:36.026 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_1_[92137]: starting. 2023-11-02T08:39:36.027 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_1_[92137]: creating pool ceph_test_rados_delete_pools_parallel.smithi151-92095 2023-11-02T08:39:36.027 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_1_[92137]: created object 0... 2023-11-02T08:39:36.027 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_1_[92137]: created object 25... 2023-11-02T08:39:36.027 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_1_[92137]: created object 49... 2023-11-02T08:39:36.028 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_1_[92137]: finishing. 2023-11-02T08:39:36.028 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_1_[92137]: shutting down. 2023-11-02T08:39:36.028 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_2_[92138]: starting. 2023-11-02T08:39:36.028 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_2_[92138]: deleting pool ceph_test_rados_delete_pools_parallel.smithi151-92095 2023-11-02T08:39:36.028 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_2_[92138]: shutting down. 2023-11-02T08:39:36.029 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: ******************************* 2023-11-02T08:39:36.029 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_3_[117875]: starting. 2023-11-02T08:39:36.029 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_3_[117875]: creating pool ceph_test_rados_delete_pools_parallel.smithi151-92095 2023-11-02T08:39:36.029 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_3_[117875]: created object 0... 2023-11-02T08:39:36.029 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_3_[117875]: created object 25... 2023-11-02T08:39:36.030 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_3_[117875]: created object 49... 2023-11-02T08:39:36.030 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_3_[117875]: finishing. 2023-11-02T08:39:36.030 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_3_[117875]: shutting down. 2023-11-02T08:39:36.030 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: ******************************* 2023-11-02T08:39:36.030 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_5_[117877]: starting. 2023-11-02T08:39:36.031 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_5_[117877]: listing objects. 2023-11-02T08:39:36.031 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_5_[117877]: listed object 0... 2023-11-02T08:39:36.031 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_5_[117877]: listed object 25... 2023-11-02T08:39:36.031 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_5_[117877]: saw 50 objects 2023-11-02T08:39:36.031 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_5_[117877]: shutting down. 2023-11-02T08:39:36.032 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: ******************************* 2023-11-02T08:39:36.032 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_4_[117876]: starting. 2023-11-02T08:39:36.032 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_4_[117876]: deleting pool ceph_test_rados_delete_pools_parallel.smithi151-92095 2023-11-02T08:39:36.032 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: process_4_[117876]: shutting down. 2023-11-02T08:39:36.032 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: ******************************* 2023-11-02T08:39:36.032 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: ******************************* 2023-11-02T08:39:36.033 INFO:tasks.workunit.client.0.smithi151.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-11-02T08:39:39.004 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: Running main() from gmock_main.cc 2023-11-02T08:39:39.004 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-11-02T08:39:39.004 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [----------] Global test environment set-up. 2023-11-02T08:39:39.004 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-11-02T08:39:39.005 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-11-02T08:39:39.005 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (22991 ms) 2023-11-02T08:39:39.005 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-11-02T08:39:39.005 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLock.LockShared (5647 ms) 2023-11-02T08:39:39.006 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-11-02T08:39:39.006 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (4303 ms) 2023-11-02T08:39:39.006 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-11-02T08:39:39.006 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (9239 ms) 2023-11-02T08:39:39.006 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-11-02T08:39:39.007 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (105 ms) 2023-11-02T08:39:39.007 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-11-02T08:39:39.007 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLock.Unlock (2279 ms) 2023-11-02T08:39:39.007 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-11-02T08:39:39.007 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (120 ms) 2023-11-02T08:39:39.008 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-11-02T08:39:39.008 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (283 ms) 2023-11-02T08:39:39.008 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [----------] 8 tests from LibRadosLock (44967 ms total) 2023-11-02T08:39:39.008 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: 2023-11-02T08:39:39.008 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-11-02T08:39:39.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-11-02T08:39:39.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (8634 ms) 2023-11-02T08:39:39.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-11-02T08:39:39.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (18924 ms) 2023-11-02T08:39:39.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-11-02T08:39:39.009 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1531 ms) 2023-11-02T08:39:39.010 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-11-02T08:39:39.010 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (5217 ms) 2023-11-02T08:39:39.010 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-11-02T08:39:39.010 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3828 ms) 2023-11-02T08:39:39.010 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-11-02T08:39:39.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (2075 ms) 2023-11-02T08:39:39.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-11-02T08:39:39.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (3721 ms) 2023-11-02T08:39:39.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-11-02T08:39:39.011 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3443 ms) 2023-11-02T08:39:39.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (47374 ms total) 2023-11-02T08:39:39.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: 2023-11-02T08:39:39.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [----------] Global test environment tear-down 2023-11-02T08:39:39.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (133038 ms total) 2023-11-02T08:39:39.012 INFO:tasks.workunit.client.0.smithi151.stdout: api_lock: [ PASSED ] 16 tests. 2023-11-02T08:39:41.675 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:04:22.566 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:41.677 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:04:22.568 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:41.755 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:04:22.646 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:41.762 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:04:22.652 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:41.778 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:22.653 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:41.780 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:22.656 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:41.781 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:04:22.671 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:41.846 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:04:22.736 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:41.858 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:22.734 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:39:46.610 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-11-02T08:39:46.611 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-11-02T08:39:46.611 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-11-02T08:39:46.611 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-11-02T08:39:46.611 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-11-02T08:39:46.611 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: notify 2023-11-02T08:39:46.612 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (14696 ms) 2023-11-02T08:39:46.612 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-11-02T08:39:46.612 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (26124 ms) 2023-11-02T08:39:46.612 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (40820 ms total) 2023-11-02T08:39:46.612 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: 2023-11-02T08:39:46.613 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-11-02T08:39:46.613 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-11-02T08:39:46.613 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: notify 2023-11-02T08:39:46.613 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (3968 ms) 2023-11-02T08:39:46.613 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-11-02T08:39:46.614 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: notify 2023-11-02T08:39:46.614 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (22159 ms) 2023-11-02T08:39:46.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-11-02T08:39:46.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (589 ms) 2023-11-02T08:39:46.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-11-02T08:39:46.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (779 ms) 2023-11-02T08:39:46.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-11-02T08:39:46.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287738000 notify_id 369367187457 notifier_gid 14077 2023-11-02T08:39:46.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5045 ms) 2023-11-02T08:39:46.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-11-02T08:39:46.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287753152 notify_id 373662154753 notifier_gid 14077 2023-11-02T08:39:46.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (792 ms) 2023-11-02T08:39:46.617 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-11-02T08:39:46.617 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287722304 notify_id 395136991234 notifier_gid 14077 2023-11-02T08:39:46.617 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (5835 ms) 2023-11-02T08:39:46.617 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-11-02T08:39:46.617 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287722304 notify_id 403726925826 notifier_gid 14077 2023-11-02T08:39:46.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (2197 ms) 2023-11-02T08:39:46.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-11-02T08:39:46.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287722304 notify_id 408021893123 notifier_gid 14077 2023-11-02T08:39:46.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (501 ms) 2023-11-02T08:39:46.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-11-02T08:39:46.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287722304 notify_id 412316860417 notifier_gid 14077 2023-11-02T08:39:46.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (1425 ms) 2023-11-02T08:39:46.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-11-02T08:39:46.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: trying... 2023-11-02T08:39:46.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287722304 notify_id 416611827715 notifier_gid 14077 2023-11-02T08:39:46.620 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: timed out 2023-11-02T08:39:48.286 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_ api_io: Running main() from gmock_main.cc 2023-11-02T08:39:48.287 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-11-02T08:39:48.287 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [----------] Global test environment set-up. 2023-11-02T08:39:48.287 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-11-02T08:39:48.287 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-11-02T08:39:48.288 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (34694 ms) 2023-11-02T08:39:48.288 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-11-02T08:39:48.288 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-11-02T08:39:48.289 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-11-02T08:39:48.289 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: no timeout :/ 2023-11-02T08:39:48.289 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: no timeout :/ 2023-11-02T08:39:48.290 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: no timeout :/ 2023-11-02T08:39:48.290 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: no timeout :/ 2023-11-02T08:39:48.290 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: no timeout :/ 2023-11-02T08:39:48.292 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (784 ms) 2023-11-02T08:39:48.292 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-11-02T08:39:48.293 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (9495 ms) 2023-11-02T08:39:48.294 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-11-02T08:39:48.294 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.Checksum (49 ms) 2023-11-02T08:39:48.294 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-11-02T08:39:48.294 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (1076 ms) 2023-11-02T08:39:48.295 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-11-02T08:39:48.295 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (261 ms) 2023-11-02T08:39:48.295 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-11-02T08:39:48.296 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (404 ms) 2023-11-02T08:39:48.296 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-11-02T08:39:48.296 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (170 ms) 2023-11-02T08:39:48.297 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-11-02T08:39:48.297 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.TruncTest (303 ms) 2023-11-02T08:39:48.297 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-11-02T08:39:48.297 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (211 ms) 2023-11-02T08:39:48.299 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-11-02T08:39:48.299 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (110 ms) 2023-11-02T08:39:48.299 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-11-02T08:39:48.300 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.RmXattr (325 ms) 2023-11-02T08:39:48.300 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-11-02T08:39:48.300 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIo.XattrIter (119 ms) 2023-11-02T08:39:48.300 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [----------] 14 tests from LibRadosIo (48003 ms total) 2023-11-02T08:39:48.301 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: 2023-11-02T08:39:48.301 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-11-02T08:39:48.301 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-11-02T08:39:48.301 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (10806 ms) 2023-11-02T08:39:48.301 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-11-02T08:39:48.302 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (12636 ms) 2023-11-02T08:39:48.302 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-11-02T08:39:48.302 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (3027 ms) 2023-11-02T08:39:48.302 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-11-02T08:39:48.303 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (1337 ms) 2023-11-02T08:39:48.303 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-11-02T08:39:48.304 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (5305 ms) 2023-11-02T08:39:48.304 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-11-02T08:39:48.304 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (407 ms) 2023-11-02T08:39:48.305 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-11-02T08:39:48.305 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (1408 ms) 2023-11-02T08:39:48.305 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-11-02T08:39:48.306 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (5938 ms) 2023-11-02T08:39:48.306 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-11-02T08:39:48.308 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (2195 ms) 2023-11-02T08:39:48.308 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-11-02T08:39:48.308 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (940 ms) 2023-11-02T08:39:48.309 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [----------] 10 tests from LibRadosIoEC (43999 ms total) 2023-11-02T08:39:48.314 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: 2023-11-02T08:39:48.315 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [----------] Global test environment tear-down 2023-11-02T08:39:48.315 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [==========] 24 tests from 2 test suites ran. (142348 ms total) 2023-11-02T08:39:48.315 INFO:tasks.workunit.client.0.smithi151.stdout: api_io: [ PASSED ] 24 tests. 2023-11-02T08:40:01.860 INFO:tasks.workunit.client.0.smithi151.stdout:pp: flushing 2023-11-02T08:40:01.860 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: flushed 2023-11-02T08:40:01.861 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3543 ms) 2023-11-02T08:40:01.861 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-11-02T08:40:01.861 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: trying... 2023-11-02T08:40:01.861 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287722304 notify_id 433791696900 notifier_gid 14077 2023-11-02T08:40:01.861 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: timed out 2023-11-02T08:40:01.862 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: flushing 2023-11-02T08:40:01.862 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: flushed 2023-11-02T08:40:01.862 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3611 ms) 2023-11-02T08:40:01.862 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-11-02T08:40:01.862 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: List watches 2023-11-02T08:40:01.863 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: notify2 2023-11-02T08:40:01.863 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287722304 notify_id 446676598789 notifier_gid 14077 2023-11-02T08:40:01.863 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: notify2 done 2023-11-02T08:40:01.863 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: watch_check 2023-11-02T08:40:01.863 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: unwatch2 2023-11-02T08:40:01.864 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: flushing 2023-11-02T08:40:01.864 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: done 2023-11-02T08:40:01.864 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3587 ms) 2023-11-02T08:40:01.864 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-11-02T08:40:01.864 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: List watches 2023-11-02T08:40:01.865 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: notify2 2023-11-02T08:40:01.865 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: handle_notify cookie 94423287722304 notify_id 459561500678 notifier_gid 14077 2023-11-02T08:40:01.865 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: notify2 done 2023-11-02T08:40:01.866 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: watch_check 2023-11-02T08:40:01.866 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: unwatch2 2023-11-02T08:40:01.866 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: flushing 2023-11-02T08:40:01.867 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: done 2023-11-02T08:40:01.867 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3228 ms) 2023-11-02T08:40:01.867 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (57260 ms total) 2023-11-02T08:40:01.867 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: 2023-11-02T08:40:01.868 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-11-02T08:40:01.868 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (155748 ms total) 2023-11-02T08:40:01.868 INFO:tasks.workunit.client.0.smithi151.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-11-02T08:40:19.120 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: Running main() from gmock_main.cc 2023-11-02T08:40:19.120 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-11-02T08:40:19.120 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] Global test environment set-up. 2023-11-02T08:40:19.121 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-11-02T08:40:19.121 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-11-02T08:40:19.121 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (13814 ms) 2023-11-02T08:40:19.121 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-11-02T08:40:19.121 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (45086 ms) 2023-11-02T08:40:19.122 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-11-02T08:40:19.122 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (22090 ms) 2023-11-02T08:40:19.122 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-11-02T08:40:19.122 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (11215 ms) 2023-11-02T08:40:19.122 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (92205 ms total) 2023-11-02T08:40:19.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: 2023-11-02T08:40:19.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-11-02T08:40:19.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-11-02T08:40:19.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (9727 ms) 2023-11-02T08:40:19.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-11-02T08:40:19.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (9348 ms) 2023-11-02T08:40:19.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-11-02T08:40:19.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (8278 ms) 2023-11-02T08:40:19.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (27353 ms total) 2023-11-02T08:40:19.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: 2023-11-02T08:40:19.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-11-02T08:40:19.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-11-02T08:40:19.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3519 ms) 2023-11-02T08:40:19.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-11-02T08:40:19.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2253 ms) 2023-11-02T08:40:19.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-11-02T08:40:19.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2134 ms) 2023-11-02T08:40:19.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-11-02T08:40:19.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2070 ms) 2023-11-02T08:40:19.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9976 ms total) 2023-11-02T08:40:19.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: 2023-11-02T08:40:19.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-11-02T08:40:19.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-11-02T08:40:19.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4889 ms) 2023-11-02T08:40:19.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-11-02T08:40:19.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4813 ms) 2023-11-02T08:40:19.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (9702 ms total) 2023-11-02T08:40:19.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: 2023-11-02T08:40:19.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [----------] Global test environment tear-down 2023-11-02T08:40:19.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (173060 ms total) 2023-11-02T08:40:19.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-11-02T08:40:19.603 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:40:19.599+0000 12a62640 -1 osd.1 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17285.0:2 82.6 82.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e102) 2023-11-02T08:40:23.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-11-02T08:40:23.149 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5991 ms) 2023-11-02T08:40:23.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-11-02T08:40:23.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (1928 ms) 2023-11-02T08:40:23.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-11-02T08:40:23.150 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (1816 ms) 2023-11-02T08:40:23.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-11-02T08:40:23.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (3626 ms) 2023-11-02T08:40:23.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-11-02T08:40:23.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4222 ms) 2023-11-02T08:40:23.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-11-02T08:40:23.151 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2297 ms) 2023-11-02T08:40:23.152 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-11-02T08:40:23.152 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3154 ms) 2023-11-02T08:40:23.152 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-11-02T08:40:23.152 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (4366 ms) 2023-11-02T08:40:23.152 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-11-02T08:40:23.153 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (205 ms) 2023-11-02T08:40:23.153 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-11-02T08:40:23.153 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (2144 ms) 2023-11-02T08:40:23.153 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-11-02T08:40:23.153 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (32389 ms) 2023-11-02T08:40:23.154 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-11-02T08:40:23.154 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (836 ms) 2023-11-02T08:40:23.154 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (86690 ms total) 2023-11-02T08:40:23.154 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: 2023-11-02T08:40:23.155 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [----------] Global test environment tear-down 2023-11-02T08:40:23.155 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (177204 ms total) 2023-11-02T08:40:23.155 INFO:tasks.workunit.client.0.smithi151.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-11-02T08:40:28.079 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_1_[92123]: starting. 2023-11-02T08:40:28.080 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_1_[92123]: creating pool ceph_test_rados_list_parallel.smithi151-92077 2023-11-02T08:40:28.080 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_1_[92123]: created object 0... 2023-11-02T08:40:28.080 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_1_[92123]: created object 25... 2023-11-02T08:40:28.080 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_1_[92123]: created object 49... 2023-11-02T08:40:28.081 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_1_[92123]: finishing. 2023-11-02T08:40:28.081 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_1_[92123]: shutting down. 2023-11-02T08:40:28.081 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_2_[92124]: starting. 2023-11-02T08:40:28.081 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_2_[92124]: listing objects. 2023-11-02T08:40:28.081 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_2_[92124]: listed object 0... 2023-11-02T08:40:28.082 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_2_[92124]: listed object 25... 2023-11-02T08:40:28.082 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_2_[92124]: saw 50 objects 2023-11-02T08:40:28.082 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_2_[92124]: shutting down. 2023-11-02T08:40:28.082 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.082 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_3_[123015]: starting. 2023-11-02T08:40:28.083 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_3_[123015]: creating pool ceph_test_rados_list_parallel.smithi151-92077 2023-11-02T08:40:28.083 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_3_[123015]: created object 0... 2023-11-02T08:40:28.083 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_3_[123015]: created object 25... 2023-11-02T08:40:28.083 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_3_[123015]: created object 49... 2023-11-02T08:40:28.084 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_3_[123015]: finishing. 2023-11-02T08:40:28.084 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_3_[123015]: shutting down. 2023-11-02T08:40:28.084 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.084 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_4_[123019]: starting. 2023-11-02T08:40:28.084 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_4_[123019]: listing objects. 2023-11-02T08:40:28.085 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_4_[123019]: listed object 0... 2023-11-02T08:40:28.085 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_4_[123019]: listed object 25... 2023-11-02T08:40:28.085 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_4_[123019]: saw 45 objects 2023-11-02T08:40:28.085 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_4_[123019]: shutting down. 2023-11-02T08:40:28.085 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.086 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_5_[123022]: starting. 2023-11-02T08:40:28.086 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_5_[123022]: removed 25 objects... 2023-11-02T08:40:28.086 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_5_[123022]: removed half of the objects 2023-11-02T08:40:28.086 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_5_[123022]: removed 50 objects... 2023-11-02T08:40:28.086 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_5_[123022]: removed 50 objects 2023-11-02T08:40:28.086 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_5_[123022]: shutting down. 2023-11-02T08:40:28.087 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.087 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.087 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_6_[131836]: starting. 2023-11-02T08:40:28.087 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_6_[131836]: creating pool ceph_test_rados_list_parallel.smithi151-92077 2023-11-02T08:40:28.088 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_6_[131836]: created object 0... 2023-11-02T08:40:28.088 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_6_[131836]: created object 25... 2023-11-02T08:40:28.088 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_6_[131836]: created object 49... 2023-11-02T08:40:28.088 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_6_[131836]: finishing. 2023-11-02T08:40:28.089 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_6_[131836]: shutting down. 2023-11-02T08:40:28.089 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.089 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.090 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_7_[131837]: starting. 2023-11-02T08:40:28.090 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_7_[131837]: listing objects. 2023-11-02T08:40:28.090 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_7_[131837]: listed object 0... 2023-11-02T08:40:28.090 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_7_[131837]: listed object 25... 2023-11-02T08:40:28.090 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_7_[131837]: listed object 50... 2023-11-02T08:40:28.090 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_7_[131837]: saw 54 objects 2023-11-02T08:40:28.091 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_7_[131837]: shutting down. 2023-11-02T08:40:28.091 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.091 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.091 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_8_[131838]: starting. 2023-11-02T08:40:28.091 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_8_[131838]: added 25 objects... 2023-11-02T08:40:28.092 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_8_[131838]: added half of the objects 2023-11-02T08:40:28.092 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_8_[131838]: added 50 objects... 2023-11-02T08:40:28.092 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_8_[131838]: added 50 objects 2023-11-02T08:40:28.092 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_8_[131838]: shutting down. 2023-11-02T08:40:28.092 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:28.093 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:40:42.059 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:05:22.949 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:40:42.065 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:05:22.956 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:40:42.115 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:22.990 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:40:42.117 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:22.992 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:40:42.121 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:05:23.011 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:40:42.122 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:05:23.013 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:40:42.198 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:05:23.088 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:40:42.238 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:23.114 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:40:42.247 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:05:23.137 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:00.468 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-11-02T08:41:00.468 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-11-02T08:41:00.468 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-11-02T08:41:00.469 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-11-02T08:41:00.469 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: seed 91733 2023-11-02T08:41:00.469 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-11-02T08:41:00.469 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (52677 ms) 2023-11-02T08:41:00.470 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-11-02T08:41:00.470 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (34785 ms) 2023-11-02T08:41:00.470 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-11-02T08:41:00.470 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (24922 ms) 2023-11-02T08:41:00.471 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-11-02T08:41:00.471 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (4872 ms) 2023-11-02T08:41:00.471 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-11-02T08:41:00.471 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (10567 ms) 2023-11-02T08:41:00.471 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (127823 ms total) 2023-11-02T08:41:00.472 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: 2023-11-02T08:41:00.472 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-11-02T08:41:00.472 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-11-02T08:41:00.472 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (7768 ms) 2023-11-02T08:41:00.472 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-11-02T08:41:00.473 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4662 ms) 2023-11-02T08:41:00.473 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-11-02T08:41:00.473 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (7667 ms) 2023-11-02T08:41:00.474 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-11-02T08:41:00.474 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (5009 ms) 2023-11-02T08:41:00.474 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-11-02T08:41:00.475 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (15116 ms) 2023-11-02T08:41:00.475 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-11-02T08:41:00.476 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.0-1159-gb5a9486b/rpm/el9/BUILD/ceph-18.2.0-1159-gb5a9486b/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-11-02T08:41:00.476 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: 2023-11-02T08:41:00.476 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-11-02T08:41:00.476 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-11-02T08:41:00.477 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi151-91733-7 2023-11-02T08:41:00.477 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: waiting for snaps to purge 2023-11-02T08:41:00.477 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (19344 ms) 2023-11-02T08:41:00.477 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (59566 ms total) 2023-11-02T08:41:00.477 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: 2023-11-02T08:41:00.478 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-11-02T08:41:00.478 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-11-02T08:41:00.478 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3250 ms) 2023-11-02T08:41:00.478 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-11-02T08:41:00.478 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2470 ms) 2023-11-02T08:41:00.479 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-11-02T08:41:00.479 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2206 ms) 2023-11-02T08:41:00.479 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-11-02T08:41:00.479 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2342 ms) 2023-11-02T08:41:00.480 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (10268 ms total) 2023-11-02T08:41:00.480 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: 2023-11-02T08:41:00.480 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-11-02T08:41:02.168 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:05:43.058 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:02.244 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:05:43.134 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:02.303 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:05:43.194 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:02.313 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:43.188 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:02.314 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:43.190 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:02.320 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:05:43.210 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:02.322 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:05:43.213 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:02.366 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:43.241 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:02.378 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:05:43.268 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:11.849 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:11.835+0000 12a62640 -1 osd.6 179 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:12.522 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:12.518+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:12.810 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:12.806+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:13.490 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:13.486+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:13.859 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:13.853+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:14.445 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:14.440+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:14.934 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:14.929+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:15.418 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:15.414+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:15.966 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:15.960+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:16.400 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:16.396+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:16.924 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:16.919+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:17.397 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:17.393+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:17.943 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:17.939+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:18.391 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:18.387+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:18.903 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:18.899+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:19.412 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:19.407+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:20.226 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:20.048+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:20.454 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:20.449+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:21.090 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:21.086+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:21.476 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:21.472+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:22.060 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:22.055+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:22.242 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:06:03.133 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:22.247 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:06:03.138 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:22.281 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:06:03.172 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:22.293 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:03.169 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:22.297 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:03.172 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:22.301 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:06:03.191 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:22.306 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:06:03.196 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:22.329 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:03.204 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:22.340 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:06:03.230 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:22.484 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:22.480+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:23.033 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:23.029+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:23.475 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:23.469+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:24.051 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:24.047+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:24.519 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:24.514+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:25.010 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:25.006+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:25.521 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:25.517+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:25.964 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:25.960+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:26.498 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:26.494+0000 12a62640 -1 osd.4 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:26.973 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:26.969+0000 12a62640 -1 osd.6 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:27.473 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:27.470+0000 12a62640 -1 osd.4 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:27.935 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:27.932+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:28.513 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:28.508+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:28.940 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:28.936+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:29.483 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:29.479+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:29.919 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:29.915+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:30.483 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:30.478+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:30.961 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:30.957+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:31.460 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:31.456+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:31.993 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:31.989+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:32.427 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:32.422+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:33.018 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:33.013+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:33.391 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:33.386+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:34.011 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:34.007+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:34.431 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:34.427+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:34.986 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:34.982+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:35.447 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:35.443+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:36.022 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:36.018+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:36.402 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:36.398+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:37.030 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:37.026+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:37.366 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:37.361+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:38.062 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:38.058+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:38.386 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:38.381+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:39.088 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:39.083+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:39.436 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:39.432+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:40.101 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:40.097+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:40.447 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:40.443+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:41.138 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:41.134+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:41.479 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:41.475+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:42.134 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:42.130+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:42.299 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:06:23.189 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:42.303 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:06:23.194 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:42.333 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:06:23.223 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:42.351 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:23.226 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:42.353 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:23.229 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:42.355 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:06:23.245 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:42.358 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:06:23.248 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:42.385 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:06:23.276 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:42.394 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:23.270 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:41:42.464 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:42.460+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:43.156 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:43.151+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:43.513 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:43.508+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:44.177 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:44.173+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:44.492 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:44.488+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:45.196 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:45.192+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:45.468 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:45.464+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:46.155 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:46.151+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:46.448 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:46.444+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:47.154 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:47.150+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:47.474 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:47.470+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:48.159 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:48.155+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:48.496 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:48.492+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:49.177 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:49.173+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:49.505 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:49.501+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:50.174 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:50.170+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:50.501 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:50.496+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:51.129 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:51.125+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:51.485 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:51.481+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:52.178 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:52.174+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:52.497 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:52.493+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:53.188 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:53.184+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:53.516 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:53.512+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:54.180 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:54.175+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:54.528 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:54.524+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:55.200 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:55.195+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:55.509 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:55.505+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:56.195 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:56.190+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:56.526 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:56.521+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:57.218 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:57.214+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:57.563 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:57.559+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:58.192 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:58.189+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:58.589 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:58.586+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:58.994 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:41:58.986+0000 12a62640 -1 osd.3 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:33.835877+0000 front 2023-11-02T08:40:33.825786+0000 (oldest deadline 2023-11-02T08:41:58.550858+0000) 2023-11-02T08:41:59.169 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:59.162+0000 12a62640 -1 osd.6 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:36.298032+0000 front 2023-11-02T08:40:36.300577+0000 (oldest deadline 2023-11-02T08:41:58.610791+0000) 2023-11-02T08:41:59.169 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:41:59.166+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:41:59.549 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:59.542+0000 12a62640 -1 osd.4 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:34.036999+0000 front 2023-11-02T08:40:34.036035+0000 (oldest deadline 2023-11-02T08:41:58.732531+0000) 2023-11-02T08:41:59.549 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:41:59.546+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:41:59.999 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:41:59.990+0000 12a62640 -1 osd.3 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:33.835877+0000 front 2023-11-02T08:40:33.825786+0000 (oldest deadline 2023-11-02T08:41:58.550858+0000) 2023-11-02T08:42:00.199 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:42:00.195+0000 12a62640 -1 osd.6 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:36.298032+0000 front 2023-11-02T08:40:36.300577+0000 (oldest deadline 2023-11-02T08:41:58.610791+0000) 2023-11-02T08:42:00.200 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:42:00.196+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:42:00.560 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:42:00.555+0000 12a62640 -1 osd.4 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:34.036999+0000 front 2023-11-02T08:40:34.036035+0000 (oldest deadline 2023-11-02T08:41:58.732531+0000) 2023-11-02T08:42:00.564 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:42:00.560+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:42:00.829 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:42:00.822+0000 12a62640 -1 osd.2 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:36.537967+0000 front 2023-11-02T08:40:36.509403+0000 (oldest deadline 2023-11-02T08:41:59.918773+0000) 2023-11-02T08:42:01.036 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:42:01.033+0000 12a62640 -1 osd.3 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:33.835877+0000 front 2023-11-02T08:40:33.825786+0000 (oldest deadline 2023-11-02T08:41:58.550858+0000) 2023-11-02T08:42:01.219 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:42:01.214+0000 12a62640 -1 osd.6 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:36.298032+0000 front 2023-11-02T08:40:36.300577+0000 (oldest deadline 2023-11-02T08:41:58.610791+0000) 2023-11-02T08:42:01.223 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:42:01.219+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:42:01.564 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:42:01.559+0000 12a62640 -1 osd.4 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:34.036999+0000 front 2023-11-02T08:40:34.036035+0000 (oldest deadline 2023-11-02T08:41:58.732531+0000) 2023-11-02T08:42:01.568 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:42:01.565+0000 12a62640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:42:01.785 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:42:01.777+0000 12a62640 -1 osd.2 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:36.537967+0000 front 2023-11-02T08:40:36.509403+0000 (oldest deadline 2023-11-02T08:41:59.918773+0000) 2023-11-02T08:42:01.869 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:42:01.861+0000 12a62640 -1 osd.7 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:36.165724+0000 front 2023-11-02T08:40:36.162393+0000 (oldest deadline 2023-11-02T08:42:01.507089+0000) 2023-11-02T08:42:01.978 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:42:01.970+0000 12a62640 -1 osd.1 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:36.063263+0000 front 2023-11-02T08:40:36.064752+0000 (oldest deadline 2023-11-02T08:42:01.465412+0000) 2023-11-02T08:42:02.006 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:42:02.003+0000 12a62640 -1 osd.3 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:33.835877+0000 front 2023-11-02T08:40:33.825786+0000 (oldest deadline 2023-11-02T08:41:58.550858+0000) 2023-11-02T08:42:02.196 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:42:02.191+0000 12a62640 -1 osd.6 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:36.298032+0000 front 2023-11-02T08:40:36.300577+0000 (oldest deadline 2023-11-02T08:41:58.610791+0000) 2023-11-02T08:42:02.196 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:42:02.192+0000 12a62640 -1 osd.6 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17279.0:41 14.d 14:b3c717c4:::4.obj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e150) 2023-11-02T08:42:02.491 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:06:43.382 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:02.502 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:43.378 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:02.505 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:06:43.395 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:02.506 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:06:43.397 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:02.510 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:06:43.401 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:02.549 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:43.424 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:02.589 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:43.464 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:02.613 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:42:02.604+0000 12a62640 -1 osd.0 182 heartbeat_check: no reply from 172.21.15.178:6822 osd.5 since back 2023-11-02T08:40:37.989785+0000 front 2023-11-02T08:40:37.784327+0000 (oldest deadline 2023-11-02T08:42:02.420505+0000) 2023-11-02T08:42:02.617 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:06:43.507 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:02.622 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:42:02.617+0000 12a62640 -1 osd.4 183 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:42:02.632 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:06:43.523 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:03.698 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:42:03.688+0000 12a62640 -1 osd.4 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17327.0:2 97.9 97.2c8c5309 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e152) 2023-11-02T08:42:08.205 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:42:08.175+0000 15267640 -1 bdev(0x18e8b0d0 /var/lib/ceph/osd/ceph-5/block) debug_aio_unlink stalled aio 0x1686a0e0 since 2023-11-02T08:41:39.050135+0000, timeout is 5s 2023-11-02T08:42:09.565 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-11-02T08:42:09.565 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4194 ms) 2023-11-02T08:42:09.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-11-02T08:42:09.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3979 ms) 2023-11-02T08:42:09.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-11-02T08:42:09.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (58475 ms) 2023-11-02T08:42:09.566 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (66648 ms total) 2023-11-02T08:42:09.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: 2023-11-02T08:42:09.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-11-02T08:42:09.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (283485 ms total) 2023-11-02T08:42:09.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-11-02T08:42:09.567 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-11-02T08:42:09.569 INFO:tasks.workunit.client.0.smithi151.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-11-02T08:42:14.808 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:42:14.803+0000 14265640 -1 osd.5 193 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:42:15.267 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-11-02T08:42:15.267 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-11-02T08:42:15.267 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-11-02T08:42:15.267 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-11-02T08:42:15.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-11-02T08:42:15.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-11-02T08:42:15.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: 2023-11-02T08:42:15.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-11-02T08:42:15.268 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: seed 91542 2023-11-02T08:42:15.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-11-02T08:42:15.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (21 ms) 2023-11-02T08:42:15.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-11-02T08:42:15.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (38068 ms) 2023-11-02T08:42:15.269 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-11-02T08:42:15.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (3673 ms) 2023-11-02T08:42:15.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-11-02T08:42:15.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (1135 ms) 2023-11-02T08:42:15.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-11-02T08:42:15.270 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (532 ms) 2023-11-02T08:42:15.271 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-11-02T08:42:15.271 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1043 ms) 2023-11-02T08:42:15.271 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-11-02T08:42:15.271 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (237 ms) 2023-11-02T08:42:15.271 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-11-02T08:42:15.272 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (180 ms) 2023-11-02T08:42:15.272 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-11-02T08:42:15.272 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (48 ms) 2023-11-02T08:42:15.272 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-11-02T08:42:15.272 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (1018 ms) 2023-11-02T08:42:15.273 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-11-02T08:42:15.273 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (135 ms) 2023-11-02T08:42:15.273 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-11-02T08:42:15.273 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (179 ms) 2023-11-02T08:42:15.275 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-11-02T08:42:15.275 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (570 ms) 2023-11-02T08:42:15.275 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-11-02T08:42:15.276 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: osd_max_attr_size = 0 2023-11-02T08:42:15.276 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-11-02T08:42:15.276 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (111912 ms) 2023-11-02T08:42:15.277 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-11-02T08:42:15.277 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (19566 ms) 2023-11-02T08:42:15.277 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-11-02T08:42:15.278 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-11-02T08:42:15.278 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: done waiting, doing copies 2023-11-02T08:42:15.278 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: waiting for final deep scrubs... 2023-11-02T08:42:15.279 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: done waiting 2023-11-02T08:42:15.279 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (78909 ms) 2023-11-02T08:42:15.280 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-11-02T08:42:15.280 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (20160 ms) 2023-11-02T08:42:15.280 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-11-02T08:42:15.282 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (800 ms) 2023-11-02T08:42:15.282 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-11-02T08:42:15.282 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5509 ms) 2023-11-02T08:42:15.282 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-11-02T08:42:15.282 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-11-02T08:42:15.283 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-11-02T08:42:15.283 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-11-02T08:42:15.283 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-11-02T08:42:15.283 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-11-02T08:42:16.368 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [------ list_parallel: ******************************* 2023-11-02T08:42:16.369 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_9_[155346]: starting. 2023-11-02T08:42:16.370 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_9_[155346]: creating pool ceph_test_rados_list_parallel.smithi151-92077 2023-11-02T08:42:16.370 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_9_[155346]: created object 0... 2023-11-02T08:42:16.370 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_9_[155346]: created object 25... 2023-11-02T08:42:16.371 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_9_[155346]: created object 49... 2023-11-02T08:42:16.371 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_9_[155346]: finishing. 2023-11-02T08:42:16.371 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_9_[155346]: shutting down. 2023-11-02T08:42:16.372 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.372 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.373 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.373 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_10_[155347]: starting. 2023-11-02T08:42:16.373 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_10_[155347]: listing objects. 2023-11-02T08:42:16.374 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_10_[155347]: listed object 0... 2023-11-02T08:42:16.374 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_10_[155347]: listed object 25... 2023-11-02T08:42:16.374 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_10_[155347]: listed object 50... 2023-11-02T08:42:16.375 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_10_[155347]: listed object 75... 2023-11-02T08:42:16.375 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_10_[155347]: saw 99 objects 2023-11-02T08:42:16.375 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_10_[155347]: shutting down. 2023-11-02T08:42:16.376 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.376 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.376 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.376 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_11_[155348]: starting. 2023-11-02T08:42:16.376 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_11_[155348]: added 25 objects... 2023-11-02T08:42:16.377 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_11_[155348]: added half of the objects 2023-11-02T08:42:16.377 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_11_[155348]: added 50 objects... 2023-11-02T08:42:16.377 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_11_[155348]: added 50 objects 2023-11-02T08:42:16.377 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_11_[155348]: shutting down. 2023-11-02T08:42:16.378 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.378 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.378 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.378 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_13_[155350]: starting. 2023-11-02T08:42:16.379 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_13_[155350]: removed 25 objects... 2023-11-02T08:42:16.379 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_13_[155350]: removed half of the objects 2023-11-02T08:42:16.379 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_13_[155350]: removed 50 objects... 2023-11-02T08:42:16.379 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_13_[155350]: removed 50 objects 2023-11-02T08:42:16.379 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_13_[155350]: shutting down. 2023-11-02T08:42:16.380 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.380 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.380 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.380 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_12_[155349]: starting. 2023-11-02T08:42:16.380 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_12_[155349]: added 25 objects... 2023-11-02T08:42:16.380 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_12_[155349]: added half of the objects 2023-11-02T08:42:16.381 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_12_[155349]: added 50 objects... 2023-11-02T08:42:16.381 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_12_[155349]: added 50 objects 2023-11-02T08:42:16.381 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_12_[155349]: shutting down. 2023-11-02T08:42:16.381 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.381 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.382 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.382 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.382 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_14_[155787]: starting. 2023-11-02T08:42:16.382 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_14_[155787]: creating pool ceph_test_rados_list_parallel.smithi151-92077 2023-11-02T08:42:16.382 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_14_[155787]: created object 0... 2023-11-02T08:42:16.383 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_14_[155787]: created object 25... 2023-11-02T08:42:16.383 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_14_[155787]: created object 49... 2023-11-02T08:42:16.383 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_14_[155787]: finishing. 2023-11-02T08:42:16.383 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_14_[155787]: shutting down. 2023-11-02T08:42:16.383 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.384 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.384 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.384 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:16.384 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: starting. 2023-11-02T08:42:16.384 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: listing objects. 2023-11-02T08:42:22.525 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:03.416 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:22.528 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:03.419 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:22.534 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:03.410 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:22.540 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:03.415 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:22.588 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:03.478 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:22.595 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:03.486 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:22.601 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:03.477 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:22.613 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:03.504 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:22.663 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:03.553 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:23.480 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: listed object 0... 2023-11-02T08:42:23.480 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: listed object 25... 2023-11-02T08:42:23.480 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: listed object 50... 2023-11-02T08:42:23.480 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: listed object 75... 2023-11-02T08:42:23.481 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: listed object 100... 2023-11-02T08:42:23.481 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: listed object 125... 2023-11-02T08:42:23.481 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: saw 150 objects 2023-11-02T08:42:23.481 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_15_[155788]: shutting down. 2023-11-02T08:42:23.482 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:23.482 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:23.482 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:23.482 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:23.482 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_16_[155789]: starting. 2023-11-02T08:42:23.483 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_16_[155789]: added 25 objects... 2023-11-02T08:42:23.483 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_16_[155789]: added half of the objects 2023-11-02T08:42:23.483 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_16_[155789]: added 50 objects... 2023-11-02T08:42:23.483 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_16_[155789]: added 50 objects 2023-11-02T08:42:23.483 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: process_16_[155789]: shutting down. 2023-11-02T08:42:23.484 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:23.484 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:23.484 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:23.484 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:23.484 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******************************* 2023-11-02T08:42:23.485 INFO:tasks.workunit.client.0.smithi151.stdout: list_parallel: ******* SUCCESS ********** 2023-11-02T08:42:27.613 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: Running main() from gmock_main.cc 2023-11-02T08:42:27.614 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-11-02T08:42:27.614 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [----------] Global test environment set-up. 2023-11-02T08:42:27.614 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-11-02T08:42:27.614 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-11-02T08:42:27.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4503 ms) 2023-11-02T08:42:27.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-11-02T08:42:27.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (24133 ms) 2023-11-02T08:42:27.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-11-02T08:42:27.615 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (4749 ms) 2023-11-02T08:42:27.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-11-02T08:42:27.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (22338 ms) 2023-11-02T08:42:27.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-11-02T08:42:27.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (34075 ms) 2023-11-02T08:42:27.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-11-02T08:42:27.616 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (25448 ms) 2023-11-02T08:42:27.617 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-11-02T08:42:27.617 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (7731 ms) 2023-11-02T08:42:27.617 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-11-02T08:42:27.617 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (5994 ms) 2023-11-02T08:42:27.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-11-02T08:42:27.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (6347 ms) 2023-11-02T08:42:27.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-11-02T08:42:27.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (5514 ms) 2023-11-02T08:42:27.618 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-11-02T08:42:27.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (34509 ms) 2023-11-02T08:42:27.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-11-02T08:42:27.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3062 ms) 2023-11-02T08:42:27.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-11-02T08:42:27.619 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3869 ms) 2023-11-02T08:42:27.620 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-11-02T08:42:27.620 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (4322 ms) 2023-11-02T08:42:27.620 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-11-02T08:42:27.620 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.Flush (4569 ms) 2023-11-02T08:42:27.620 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-11-02T08:42:27.620 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3306 ms) 2023-11-02T08:42:27.621 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-11-02T08:42:27.621 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3188 ms) 2023-11-02T08:42:27.621 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-11-02T08:42:27.621 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3253 ms) 2023-11-02T08:42:27.621 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-11-02T08:42:27.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2468 ms) 2023-11-02T08:42:27.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-11-02T08:42:27.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (4542 ms) 2023-11-02T08:42:27.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-11-02T08:42:27.622 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (70940 ms) 2023-11-02T08:42:27.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-11-02T08:42:27.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4783 ms) 2023-11-02T08:42:27.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-11-02T08:42:27.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4558 ms) 2023-11-02T08:42:27.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-11-02T08:42:27.623 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4595 ms) 2023-11-02T08:42:27.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-11-02T08:42:27.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4707 ms) 2023-11-02T08:42:27.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-11-02T08:42:27.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4183 ms) 2023-11-02T08:42:27.624 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [----------] 26 tests from LibRadosAio (301688 ms total) 2023-11-02T08:42:27.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: 2023-11-02T08:42:27.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-11-02T08:42:42.847 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:23.737 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:42.848 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:23.724 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:42.849 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:23.740 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:42.856 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:23.732 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:42.920 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:23.811 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:42.922 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:23.812 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:42.924 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:23.815 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:42.938 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:23.813 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:42.996 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:23.886 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:47.410 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.Simple----] 22 tests from LibRadosMiscPP (283696 ms total) 2023-11-02T08:42:47.411 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: 2023-11-02T08:42:47.411 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-11-02T08:42:47.411 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-11-02T08:42:47.412 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (4273 ms) 2023-11-02T08:42:47.412 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (4273 ms total) 2023-11-02T08:42:47.412 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: 2023-11-02T08:42:47.412 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-11-02T08:42:47.412 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-11-02T08:42:47.412 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (792 ms) 2023-11-02T08:42:47.413 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-11-02T08:42:47.413 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (67 ms) 2023-11-02T08:42:47.413 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (859 ms total) 2023-11-02T08:42:47.413 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: 2023-11-02T08:42:47.413 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-11-02T08:42:47.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-11-02T08:42:47.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (354 ms) 2023-11-02T08:42:47.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-11-02T08:42:47.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (2571 ms) 2023-11-02T08:42:47.414 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (2926 ms total) 2023-11-02T08:42:47.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: 2023-11-02T08:42:47.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-11-02T08:42:47.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-11-02T08:42:47.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (807 ms) 2023-11-02T08:42:47.415 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-11-02T08:42:47.416 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (43 ms) 2023-11-02T08:42:47.416 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (850 ms total) 2023-11-02T08:42:47.416 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: 2023-11-02T08:42:47.416 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-11-02T08:42:47.416 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-11-02T08:42:47.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1908 ms) 2023-11-02T08:42:47.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1908 ms total) 2023-11-02T08:42:47.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: 2023-11-02T08:42:47.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-11-02T08:42:47.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (321384 ms total) 2023-11-02T08:42:47.417 INFO:tasks.workunit.client.0.smithi151.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-11-02T08:42:51.623 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:32.513 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:51.633 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:32.524 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:51.645 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:32.520 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:51.646 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:32.522 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:51.659 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:32.549 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:51.710 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:32.600 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:51.748 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:32.624 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:51.754 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:32.645 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:51.759 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:32.650 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:42:55.121 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: Running main() from gmock_main.cc 2023-11-02T08:42:55.122 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-11-02T08:42:55.122 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-11-02T08:42:55.122 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-11-02T08:42:55.122 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-11-02T08:42:55.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4517 ms) 2023-11-02T08:42:55.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-11-02T08:42:55.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (93861 ms) 2023-11-02T08:42:55.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-11-02T08:42:55.123 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (24596 ms) 2023-11-02T08:42:55.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-11-02T08:42:55.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (5976 ms) 2023-11-02T08:42:55.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-11-02T08:42:55.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (8594 ms) 2023-11-02T08:42:55.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-11-02T08:42:55.124 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (4672 ms) 2023-11-02T08:42:55.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-11-02T08:42:55.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (33201 ms) 2023-11-02T08:42:55.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-11-02T08:42:55.125 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2930 ms) 2023-11-02T08:42:55.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-11-02T08:42:55.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3924 ms) 2023-11-02T08:42:55.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-11-02T08:42:55.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3159 ms) 2023-11-02T08:42:55.126 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-11-02T08:42:55.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3187 ms) 2023-11-02T08:42:55.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-11-02T08:42:55.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (4669 ms) 2023-11-02T08:42:55.127 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-11-02T08:42:55.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (85575 ms) 2023-11-02T08:42:55.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-11-02T08:42:55.128 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3746 ms) 2023-11-02T08:42:55.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-11-02T08:42:55.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3399 ms) 2023-11-02T08:42:55.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-11-02T08:42:55.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3332 ms) 2023-11-02T08:42:55.129 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-11-02T08:42:55.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (5772 ms) 2023-11-02T08:42:55.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-11-02T08:42:55.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4493 ms) 2023-11-02T08:42:55.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-11-02T08:42:55.130 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3340 ms) 2023-11-02T08:42:55.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-11-02T08:42:55.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3095 ms) 2023-11-02T08:42:55.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-11-02T08:42:55.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2859 ms) 2023-11-02T08:42:55.131 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-11-02T08:42:55.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (4248 ms) 2023-11-02T08:42:55.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-11-02T08:42:55.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3224 ms) 2023-11-02T08:42:55.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-11-02T08:42:55.132 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2730 ms) 2023-11-02T08:42:55.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-11-02T08:42:55.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3390 ms) 2023-11-02T08:42:55.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-11-02T08:42:55.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3320 ms) 2023-11-02T08:42:55.133 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-11-02T08:43:02.737 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:43.627 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:02.739 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:43.614 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:02.739 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:43.630 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:02.743 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:43.618 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:02.771 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:07:43.662 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:02.775 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:43.651 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:02.790 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:43.681 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:02.793 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:43.684 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:02.822 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:07:43.712 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:22.926 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:08:03.816 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:22.981 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:03.857 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:22.982 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:03.858 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:22.985 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:08:03.875 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:22.987 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:08:03.878 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:22.990 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:08:03.880 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:23.026 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:03.901 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:23.048 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:08:03.939 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:23.052 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:08:03.942 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:42.910 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:08:23.800 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:42.960 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:23.836 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:42.962 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:08:23.853 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:42.963 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:23.839 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:42.988 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:23.863 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:42.999 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:08:23.890 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:43.028 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:08:23.919 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:43.031 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:08:23.921 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:43.061 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:08:23.951 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:43:43.659 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRados api_list: : cursor()=9:f905c69b:::2:head expected=9:f905c69b:::2:head 2023-11-02T08:43:43.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:f905c69b:::2:head -> 2 2023-11-02T08:43:43.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=2 expected=2 2023-11-02T08:43:43.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:cfc208b3:::3:head 2023-11-02T08:43:43.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:cfc208b3:::3:head expected=9:cfc208b3:::3:head 2023-11-02T08:43:43.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2023-11-02T08:43:43.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=3 expected=3 2023-11-02T08:43:43.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2023-11-02T08:43:43.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:c4fdafeb:::6:head expected=9:c4fdafeb:::6:head 2023-11-02T08:43:43.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2023-11-02T08:43:43.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=6 expected=6 2023-11-02T08:43:43.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2023-11-02T08:43:43.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:bd63b0f1:::8:head expected=9:bd63b0f1:::8:head 2023-11-02T08:43:43.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2023-11-02T08:43:43.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=8 expected=8 2023-11-02T08:43:43.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:b29083e3:::5:head 2023-11-02T08:43:43.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:b29083e3:::5:head expected=9:b29083e3:::5:head 2023-11-02T08:43:43.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:b29083e3:::5:head -> 5 2023-11-02T08:43:43.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=5 expected=5 2023-11-02T08:43:43.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:89d3ae78:::11:head 2023-11-02T08:43:43.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:89d3ae78:::11:head expected=9:89d3ae78:::11:head 2023-11-02T08:43:43.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2023-11-02T08:43:43.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=11 expected=11 2023-11-02T08:43:43.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:863748b0:::15:head 2023-11-02T08:43:43.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:863748b0:::15:head expected=9:863748b0:::15:head 2023-11-02T08:43:43.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:863748b0:::15:head -> 15 2023-11-02T08:43:43.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=15 expected=15 2023-11-02T08:43:43.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : seek to 9:62a1935d:::14:head 2023-11-02T08:43:43.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : cursor()=9:62a1935d:::14:head expected=9:62a1935d:::14:head 2023-11-02T08:43:43.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: > 9:62a1935d:::14:head -> 14 2023-11-02T08:43:43.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: : entry=14 expected=14 2023-11-02T08:43:43.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (33005 ms) 2023-11-02T08:43:43.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-11-02T08:43:43.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (33053 ms) 2023-11-02T08:43:43.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-11-02T08:43:43.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: split 0/5 -> MIN 9:33333333::::head 2023-11-02T08:43:43.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: split 1/5 -> 9:33333333::::head 9:66666666::::head 2023-11-02T08:43:43.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: split 2/5 -> 9:66666666::::head 9:99999999::::head 2023-11-02T08:43:43.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: split 3/5 -> 9:99999999::::head 9:cccccccc::::head 2023-11-02T08:43:43.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: split 4/5 -> 9:cccccccc::::head MAX 2023-11-02T08:43:43.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (198503 ms) 2023-11-02T08:43:43.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [----------] 7 tests from LibRadosList (338678 ms total) 2023-11-02T08:43:43.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 2023-11-02T08:43:43.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-11-02T08:43:43.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-11-02T08:43:43.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (3997 ms) 2023-11-02T08:43:43.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-11-02T08:43:43.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: myset foo1,foo2,foo3 2023-11-02T08:43:43.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo1 2023-11-02T08:43:43.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo2 2023-11-02T08:43:43.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo3 2023-11-02T08:43:43.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: myset foo1,foo4,foo5 2023-11-02T08:43:43.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo4 2023-11-02T08:43:43.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo5 2023-11-02T08:43:43.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo1 2023-11-02T08:43:43.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: myset foo6,foo7 2023-11-02T08:43:43.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo7 2023-11-02T08:43:43.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: foo6 2023-11-02T08:43:43.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-11-02T08:43:43.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns1:foo4 2023-11-02T08:43:43.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns1:foo5 2023-11-02T08:43:43.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns2:foo7 2023-11-02T08:43:43.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns2:foo6 2023-11-02T08:43:43.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: ns1:foo1 2023-11-02T08:43:43.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: :foo1 2023-11-02T08:43:43.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: :foo2 2023-11-02T08:43:43.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: :foo3 2023-11-02T08:43:43.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (26748 ms) 2023-11-02T08:43:43.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-11-02T08:43:43.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 1 0 2023-11-02T08:43:43.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 10 0 2023-11-02T08:43:43.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 13 0 2023-11-02T08:43:51.773 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 7 0 2023-11-02T08:43:51.774 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 14 0 2023-11-02T08:43:51.774 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 0 0 2023-11-02T08:43:51.774 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 15 0 2023-11-02T08:43:51.775 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 11 0 2023-11-02T08:43:51.775 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 5 0 2023-11-02T08:43:51.775 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 8 0 2023-11-02T08:43:51.775 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 6 0 2023-11-02T08:43:51.776 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 3 0 2023-11-02T08:43:51.776 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 4 0 2023-11-02T08:43:51.776 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 12 0 2023-11-02T08:43:51.776 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 9 0 2023-11-02T08:43:51.776 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 2 0 2023-11-02T08:43:51.777 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-11-02T08:43:51.777 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3382 ms) 2023-11-02T08:43:51.777 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [----------] 3 tests from LibRadosListEC (34127 ms total) 2023-11-02T08:43:51.778 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 2023-11-02T08:43:51.779 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-11-02T08:43:51.779 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-11-02T08:43:51.779 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3424 ms) 2023-11-02T08:43:51.780 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [----------] 1 test from LibRadosListNP (3424 ms total) 2023-11-02T08:43:51.780 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: 2023-11-02T08:43:51.780 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [----------] Global test environment tear-down 2023-11-02T08:43:51.781 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [==========] 11 tests from 3 test suites ran. (385770 ms total) 2023-11-02T08:43:51.781 INFO:tasks.workunit.client.0.smithi151.stdout: api_list: [ PASSED ] 11 tests. 2023-11-02T08:44:03.357 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:08:44.247 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:03.364 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:44.239 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:03.372 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:44.247 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:03.373 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:08:44.263 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:03.396 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:08:44.286 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:03.431 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:08:44.321 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:03.445 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:08:44.336 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:03.468 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:44.344 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:03.521 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:08:44.411 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:23.252 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:04.141 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:23.309 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:04.199 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:23.314 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:04.189 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:23.322 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:04.198 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:23.362 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:04.252 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:23.377 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:04.267 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:23.382 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:04.272 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:23.386 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:04.261 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:23.435 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:04.325 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:43.343 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:24.233 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:43.412 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:24.303 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:43.446 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:24.336 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:43.447 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:24.337 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:43.472 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:24.362 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:43.501 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:24.391 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:43.529 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:24.405 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:43.531 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:24.407 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:43.597 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:24.472 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:47.619 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:28.510 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:47.664 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:28.539 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:47.693 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:28.584 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:47.697 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:28.572 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:47.726 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:28.617 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:47.729 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:28.605 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:47.761 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:28.652 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:47.767 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:28.658 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:47.792 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:28.683 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:44:51.669 INFO:tasks.workunit.client.0.smithi151.stdout:Write 2023-11-02T08:44:51.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (11449 ms) 2023-11-02T08:44:51.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-11-02T08:44:51.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7103 ms) 2023-11-02T08:44:51.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-11-02T08:44:51.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7745 ms) 2023-11-02T08:44:51.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-11-02T08:44:51.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7049 ms) 2023-11-02T08:44:51.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-11-02T08:44:51.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (9185 ms) 2023-11-02T08:44:51.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-11-02T08:44:51.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7873 ms) 2023-11-02T08:44:51.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-11-02T08:44:51.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (25810 ms) 2023-11-02T08:44:51.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-11-02T08:44:51.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7958 ms) 2023-11-02T08:44:51.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-11-02T08:44:51.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (8008 ms) 2023-11-02T08:44:51.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-11-02T08:44:51.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7941 ms) 2023-11-02T08:44:51.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-11-02T08:44:51.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6632 ms) 2023-11-02T08:44:51.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-11-02T08:44:51.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7574 ms) 2023-11-02T08:44:51.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-11-02T08:44:51.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6894 ms) 2023-11-02T08:44:51.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-11-02T08:44:51.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7754 ms) 2023-11-02T08:44:51.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-11-02T08:44:51.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6745 ms) 2023-11-02T08:44:51.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-11-02T08:44:51.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (8332 ms) 2023-11-02T08:44:51.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (144053 ms total) 2023-11-02T08:44:51.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: 2023-11-02T08:44:51.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [----------] Global test environment tear-down 2023-11-02T08:44:51.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (445741 ms total) 2023-11-02T08:44:51.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio: [ PASSED ] 42 tests. 2023-11-02T08:45:03.541 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:44.432 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:03.544 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:44.435 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:03.587 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:09:44.477 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:03.594 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:44.470 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:03.597 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:44.472 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:03.598 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:44.489 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:03.603 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:44.493 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:03.635 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:44.511 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:03.646 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:09:44.537 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:23.562 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:10:04.452 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:23.565 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:10:04.455 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:23.601 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:10:04.492 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:23.611 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:04.487 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:23.612 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:04.488 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:23.624 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:10:04.515 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:23.628 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:10:04.516 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:23.647 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:04.523 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:23.661 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:10:04.552 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:43.773 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:10:24.663 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:43.777 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:10:24.667 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:43.823 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:24.699 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:43.824 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:24.700 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:43.838 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:10:24.729 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:43.853 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:10:24.743 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:43.854 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:10:24.744 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:43.861 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:24.736 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:45:43.893 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:10:24.784 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:03.676 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:10:44.566 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:03.724 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:10:44.615 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:03.750 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:10:44.640 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:03.775 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:44.650 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:03.777 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:10:44.667 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:03.779 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:44.655 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:03.780 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:10:44.671 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:03.803 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:44.678 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:03.806 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:10:44.696 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:23.758 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:04.648 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:23.762 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:04.653 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:23.782 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:04.672 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:23.810 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:04.685 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:23.813 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:04.703 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:23.813 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:04.688 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:23.816 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:04.706 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:23.829 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:04.705 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:23.833 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:04.724 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:43.807 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:24.698 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:43.810 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:24.700 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:43.828 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:24.719 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:43.857 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:24.733 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:43.859 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:24.735 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:43.864 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:24.754 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:43.867 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:24.757 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:43.875 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:24.751 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:43.885 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:24.775 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:46:50.210 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:46:50.189+0000 12a62640 -1 osd.7 384 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22075.0:1 181.5s0 181.8e3145ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384) 2023-11-02T08:46:51.169 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:46:51.164+0000 12a62640 -1 osd.7 387 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22075.0:1 181.5s0 181.8e3145ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384) 2023-11-02T08:47:03.826 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:44.701 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:03.829 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:44.705 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:03.833 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:44.724 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:03.842 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:44.733 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:03.864 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:44.739 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:03.898 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:44.788 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:03.898 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:44.789 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:03.905 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:44.796 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:03.944 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:44.834 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:04.658 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:45.534 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:04.659 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:45.535 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:04.668 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:45.543 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:04.885 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:45.776 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:04.887 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:45.777 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:04.901 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:11:45.792 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:04.934 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:45.825 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:05.000 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:45.891 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:05.023 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:11:45.913 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:08.369 INFO:tasks.workunit.client.0.smithi151.stdout:Aio.RoundTripAppendPP (3385 ms) 2023-11-02T08:47:08.370 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-11-02T08:47:08.370 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3662 ms) 2023-11-02T08:47:08.370 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-11-02T08:47:08.370 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3177 ms) 2023-11-02T08:47:08.371 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-11-02T08:47:08.371 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3609 ms) 2023-11-02T08:47:08.371 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-11-02T08:47:08.371 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: setting pool EIO 2023-11-02T08:47:08.371 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: waiting for 314 2023-11-02T08:47:08.372 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: max_success 0, min_failed 1 2023-11-02T08:47:08.372 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (7658 ms) 2023-11-02T08:47:08.372 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-11-02T08:47:08.372 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (29215 ms) 2023-11-02T08:47:08.372 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (376515 ms total) 2023-11-02T08:47:08.373 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: 2023-11-02T08:47:08.373 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-11-02T08:47:08.373 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-11-02T08:47:08.373 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3574 ms) 2023-11-02T08:47:08.373 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-11-02T08:47:08.373 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10378 ms) 2023-11-02T08:47:08.374 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-11-02T08:47:08.374 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16616 ms) 2023-11-02T08:47:08.374 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-11-02T08:47:08.374 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3349 ms) 2023-11-02T08:47:08.374 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (33917 ms total) 2023-11-02T08:47:08.375 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: 2023-11-02T08:47:08.375 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-11-02T08:47:08.377 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-11-02T08:47:08.377 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3281 ms) 2023-11-02T08:47:08.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3281 ms total) 2023-11-02T08:47:08.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: 2023-11-02T08:47:08.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-11-02T08:47:08.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-11-02T08:47:08.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14754 ms) 2023-11-02T08:47:08.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-11-02T08:47:08.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7441 ms) 2023-11-02T08:47:08.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-11-02T08:47:08.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6777 ms) 2023-11-02T08:47:08.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-11-02T08:47:08.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7074 ms) 2023-11-02T08:47:08.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-11-02T08:47:08.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3741 ms) 2023-11-02T08:47:08.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-11-02T08:47:08.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7467 ms) 2023-11-02T08:47:08.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-11-02T08:47:08.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7802 ms) 2023-11-02T08:47:08.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-11-02T08:47:08.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (15167 ms) 2023-11-02T08:47:08.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-11-02T08:47:08.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7135 ms) 2023-11-02T08:47:08.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-11-02T08:47:08.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7747 ms) 2023-11-02T08:47:08.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-11-02T08:47:08.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (69297 ms) 2023-11-02T08:47:08.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-11-02T08:47:08.383 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7534 ms) 2023-11-02T08:47:08.383 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-11-02T08:47:08.383 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6793 ms) 2023-11-02T08:47:24.143 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:05.033 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:24.208 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:05.098 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:24.259 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:05.149 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:24.269 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:05.144 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:24.273 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:05.163 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:24.273 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:05.149 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:24.275 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:05.165 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:24.316 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:05.191 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:24.346 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:05.236 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:39.097 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:19.987 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:39.142 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:20.018 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:39.143 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:20.019 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:39.155 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:20.046 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:39.178 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:20.053 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:39.207 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:20.097 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:39.210 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:20.101 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:39.216 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:20.106 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:39.242 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:20.132 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:49.263 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:30.153 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:49.315 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:30.205 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:49.316 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:30.191 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:49.320 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:30.195 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:49.350 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:30.225 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:49.352 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:30.242 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:49.378 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:30.268 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:49.379 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:30.269 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:49.408 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:30.299 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.214 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:35.104 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.219 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:35.109 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.235 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:35.125 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.267 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:35.142 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.271 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:35.146 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.284 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:35.160 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.289 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:35.179 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.292 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:35.182 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.308 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:35.198 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:47:54.978 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-11-02T08:47:54.979 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2983 ms) 2023-11-02T08:47:54.979 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-11-02T08:47:54.979 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6977 ms) 2023-11-02T08:47:54.980 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-11-02T08:47:54.980 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6869 ms) 2023-11-02T08:47:54.980 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-11-02T08:47:54.980 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7483 ms) 2023-11-02T08:47:54.980 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-11-02T08:47:54.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7202 ms) 2023-11-02T08:47:54.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-11-02T08:47:54.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7928 ms) 2023-11-02T08:47:54.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-11-02T08:47:54.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7165 ms) 2023-11-02T08:47:54.981 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (215336 ms total) 2023-11-02T08:47:54.982 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: 2023-11-02T08:47:54.982 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-11-02T08:47:54.982 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (629049 ms total) 2023-11-02T08:47:54.982 INFO:tasks.workunit.client.0.smithi151.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-11-02T08:48:09.472 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:50.362 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:09.479 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:50.369 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:09.507 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:12:50.398 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:09.525 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:50.415 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:09.526 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:50.401 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:09.526 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:50.402 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:09.528 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:50.419 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:09.552 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:50.428 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:09.559 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:12:50.449 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:19.536 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:00.427 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:19.542 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:00.432 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:19.563 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:00.454 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:19.587 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:00.462 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:19.588 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:00.463 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:19.591 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:00.482 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:19.596 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:00.486 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:19.609 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:00.484 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:19.618 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:00.508 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:29.594 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:10.485 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:29.601 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:10.490 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:29.605 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:10.481 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:29.609 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:10.485 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:29.624 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:10.515 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:29.635 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:10.510 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:29.651 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:10.541 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:29.652 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:10.542 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:29.687 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:10.577 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:34.522 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:15.412 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:34.576 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:15.451 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:34.577 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:15.452 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:34.581 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:15.472 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:34.592 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:15.468 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:34.600 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:15.491 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:34.632 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:15.523 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:34.634 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:15.524 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:34.648 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:15.539 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:49.713 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:30.603 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:49.762 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:30.652 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:49.790 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:30.680 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:49.810 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:30.685 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:49.811 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:30.686 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:49.813 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:30.704 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:49.814 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:30.705 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:49.838 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:30.713 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:49.844 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:30.734 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:48:59.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-11-02T08:48:59.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-11-02T08:48:59.625 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-11-02T08:48:59.626 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: seed 91630 2023-11-02T08:48:59.626 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-11-02T08:48:59.626 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (42458 ms) 2023-11-02T08:48:59.626 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-11-02T08:48:59.626 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (79782 ms) 2023-11-02T08:48:59.627 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-11-02T08:48:59.627 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (834 ms) 2023-11-02T08:48:59.627 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (123074 ms total) 2023-11-02T08:48:59.627 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: 2023-11-02T08:48:59.627 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-11-02T08:48:59.628 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-11-02T08:48:59.628 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (48560 ms) 2023-11-02T08:48:59.628 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-11-02T08:48:59.628 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (99510 ms) 2023-11-02T08:48:59.628 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-11-02T08:48:59.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (20744 ms) 2023-11-02T08:48:59.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-11-02T08:48:59.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: my_snaps [4] 2023-11-02T08:48:59.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: my_snaps [5,4] 2023-11-02T08:48:59.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: my_snaps [6,5,4] 2023-11-02T08:48:59.629 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-11-02T08:48:59.630 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: promoting some heads 2023-11-02T08:48:59.630 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: promoting from clones for snap 7 2023-11-02T08:48:59.630 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: promoting from clones for snap 6 2023-11-02T08:48:59.630 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: promoting from clones for snap 5 2023-11-02T08:48:59.630 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: promoting from clones for snap 4 2023-11-02T08:48:59.631 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: waiting for scrubs... 2023-11-02T08:48:59.631 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: done waiting 2023-11-02T08:48:59.631 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (150148 ms) 2023-11-02T08:48:59.631 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-11-02T08:48:59.631 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (11211 ms) 2023-11-02T08:48:59.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-11-02T08:48:59.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (23949 ms) 2023-11-02T08:48:59.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-11-02T08:48:59.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8282 ms) 2023-11-02T08:48:59.632 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-11-02T08:48:59.633 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (72998 ms) 2023-11-02T08:48:59.633 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-11-02T08:48:59.633 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11468 ms) 2023-11-02T08:48:59.633 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-11-02T08:48:59.634 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9160 ms) 2023-11-02T08:48:59.634 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-11-02T08:48:59.634 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9928 ms) 2023-11-02T08:48:59.634 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-11-02T08:48:59.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (40431 ms) 2023-11-02T08:48:59.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-11-02T08:48:59.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8885 ms) 2023-11-02T08:48:59.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-11-02T08:48:59.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (9577 ms) 2023-11-02T08:48:59.635 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-11-02T08:48:59.636 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (15419 ms) 2023-11-02T08:48:59.636 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-11-02T08:48:59.636 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (9729 ms) 2023-11-02T08:48:59.636 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-11-02T08:48:59.636 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (12910 ms) 2023-11-02T08:49:04.687 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:45.578 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:04.738 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:45.629 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:04.782 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:45.658 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:04.784 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:45.675 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:04.787 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:45.663 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:04.805 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:45.696 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:04.806 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:45.697 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:04.817 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:45.692 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:04.836 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:45.727 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:09.680 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:50.571 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:09.732 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:50.607 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:09.733 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:50.609 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:09.743 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:50.633 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:09.752 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:50.643 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:09.756 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:50.646 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:09.756 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:13:50.631 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:09.788 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:13:50.679 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:09.797 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:13:50.687 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:20.645 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-11-02T08:49:20.645 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: hmm, no HitSet yet 2023-11-02T08:49:20.645 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: ok, hit_set contains 200:602f83fe:::foo:head 2023-11-02T08:49:20.645 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8850 ms) 2023-11-02T08:49:20.646 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-11-02T08:49:20.646 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg_num = 32 2023-11-02T08:49:20.646 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 0 ls 1698914956,0 2023-11-02T08:49:20.646 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 1 ls 1698914956,0 2023-11-02T08:49:20.646 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 2 ls 1698914956,0 2023-11-02T08:49:20.647 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 3 ls 1698914955,0 2023-11-02T08:49:20.647 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 4 ls 1698914956,0 2023-11-02T08:49:20.647 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 5 ls 1698914956,0 2023-11-02T08:49:20.647 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 6 ls 1698914956,0 2023-11-02T08:49:20.647 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 7 ls 1698914956,0 2023-11-02T08:49:20.648 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 8 ls 1698914956,0 2023-11-02T08:49:20.648 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 9 ls 1698914956,0 2023-11-02T08:49:20.648 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 10 ls 1698914956,0 2023-11-02T08:49:20.648 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 11 ls 1698914956,0 2023-11-02T08:49:20.648 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 12 ls 1698914956,0 2023-11-02T08:49:20.649 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 13 ls 1698914956,0 2023-11-02T08:49:20.649 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 14 ls 1698914956,0 2023-11-02T08:49:20.649 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 15 ls 1698914956,0 2023-11-02T08:49:20.649 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 16 ls 1698914955,0 2023-11-02T08:49:20.649 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 17 ls 1698914956,0 2023-11-02T08:49:20.650 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 18 ls 1698914956,0 2023-11-02T08:49:20.650 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 19 ls 1698914956,0 2023-11-02T08:49:20.650 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 20 ls 1698914956,0 2023-11-02T08:49:20.650 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 21 ls 1698914956,0 2023-11-02T08:49:20.650 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 22 ls 1698914956,0 2023-11-02T08:49:20.651 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 23 ls 1698914956,0 2023-11-02T08:49:20.651 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 24 ls 1698914956,0 2023-11-02T08:49:20.651 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 25 ls 1698914956,0 2023-11-02T08:49:20.651 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 26 ls 1698914956,0 2023-11-02T08:49:20.651 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 27 ls 1698914956,0 2023-11-02T08:49:20.652 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 28 ls 1698914956,0 2023-11-02T08:49:20.652 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 29 ls 1698914956,0 2023-11-02T08:49:20.652 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 30 ls 1698914956,0 2023-11-02T08:49:20.652 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg 31 ls 1698914956,0 2023-11-02T08:49:20.652 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: pg_num = 32 2023-11-02T08:49:20.653 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:6cac518f:::0:head 2023-11-02T08:49:20.653 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:02547ec2:::1:head 2023-11-02T08:49:20.653 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:f905c69b:::2:head 2023-11-02T08:49:20.653 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:cfc208b3:::3:head 2023-11-02T08:49:20.653 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d83876eb:::4:head 2023-11-02T08:49:20.653 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:b29083e3:::5:head 2023-11-02T08:49:20.654 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c4fdafeb:::6:head 2023-11-02T08:49:20.654 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:5c6b0b28:::7:head 2023-11-02T08:49:20.654 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:bd63b0f1:::8:head 2023-11-02T08:49:20.654 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:e960b815:::9:head 2023-11-02T08:49:20.654 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:52ea6a34:::10:head 2023-11-02T08:49:20.655 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:89d3ae78:::11:head 2023-11-02T08:49:20.655 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:de5d7c5f:::12:head 2023-11-02T08:49:20.655 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:566253c9:::13:head 2023-11-02T08:49:20.655 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:62a1935d:::14:head 2023-11-02T08:49:20.655 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:863748b0:::15:head 2023-11-02T08:49:20.656 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:3958e169:::16:head 2023-11-02T08:49:20.656 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:4d4dabf9:::17:head 2023-11-02T08:49:20.656 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:8391935d:::18:head 2023-11-02T08:49:20.656 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:28883081:::19:head 2023-11-02T08:49:20.656 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:69259c59:::20:head 2023-11-02T08:49:20.656 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:4bdb80b7:::21:head 2023-11-02T08:49:20.657 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:a11c5d71:::22:head 2023-11-02T08:49:20.657 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:271af37b:::23:head 2023-11-02T08:49:20.657 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:95b121be:::24:head 2023-11-02T08:49:20.657 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:58d1031b:::25:head 2023-11-02T08:49:20.657 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:0a050783:::26:head 2023-11-02T08:49:20.658 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c709704c:::27:head 2023-11-02T08:49:20.658 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:cbe56eaf:::28:head 2023-11-02T08:49:20.658 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:86b4b162:::29:head 2023-11-02T08:49:20.658 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:70d89383:::30:head 2023-11-02T08:49:20.658 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:dd450c7c:::31:head 2023-11-02T08:49:20.659 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:6d5729b1:::32:head 2023-11-02T08:49:20.659 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c388f3fb:::33:head 2023-11-02T08:49:20.659 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:56cfea31:::34:head 2023-11-02T08:49:20.659 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:9dbc1bf7:::35:head 2023-11-02T08:49:20.659 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:40b74ccd:::36:head 2023-11-02T08:49:20.659 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:4d5aaf42:::37:head 2023-11-02T08:49:20.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:920f362c:::38:head 2023-11-02T08:49:20.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:6cc53222:::39:head 2023-11-02T08:49:20.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:9cad833f:::40:head 2023-11-02T08:49:20.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:1ea84d41:::41:head 2023-11-02T08:49:20.660 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c4480ef6:::42:head 2023-11-02T08:49:20.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:a694361e:::43:head 2023-11-02T08:49:20.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d1bd33e9:::44:head 2023-11-02T08:49:20.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:ddc2cd5d:::45:head 2023-11-02T08:49:20.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:2b782207:::46:head 2023-11-02T08:49:20.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:7b187fca:::47:head 2023-11-02T08:49:20.661 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:90ecdf6f:::48:head 2023-11-02T08:49:20.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:a5ed95fe:::49:head 2023-11-02T08:49:20.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:ea0eaa55:::50:head 2023-11-02T08:49:20.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:f33ef17b:::51:head 2023-11-02T08:49:20.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:a0d1b2f6:::52:head 2023-11-02T08:49:20.662 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:60c5229e:::53:head 2023-11-02T08:49:20.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:edcbc575:::54:head 2023-11-02T08:49:20.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:102cf253:::55:head 2023-11-02T08:49:20.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:efb7fb0b:::56:head 2023-11-02T08:49:20.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:50d0a326:::57:head 2023-11-02T08:49:20.663 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d4dc5daf:::58:head 2023-11-02T08:49:20.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:3a130462:::59:head 2023-11-02T08:49:20.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:ec87ed71:::60:head 2023-11-02T08:49:20.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d5bc9454:::61:head 2023-11-02T08:49:20.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:3ddfe313:::62:head 2023-11-02T08:49:20.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:7c2816b9:::63:head 2023-11-02T08:49:20.664 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:47e00e4d:::64:head 2023-11-02T08:49:20.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c6410c18:::65:head 2023-11-02T08:49:20.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:b48ed237:::66:head 2023-11-02T08:49:20.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:cd63ad31:::67:head 2023-11-02T08:49:20.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:b179e92b:::68:head 2023-11-02T08:49:20.665 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:0d9f741a:::69:head 2023-11-02T08:49:20.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:6d3352ae:::70:head 2023-11-02T08:49:20.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c6d5c19e:::71:head 2023-11-02T08:49:20.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:bc4729c3:::72:head 2023-11-02T08:49:20.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:77e930b9:::73:head 2023-11-02T08:49:20.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:0abeecfd:::74:head 2023-11-02T08:49:20.666 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:b7c37e15:::75:head 2023-11-02T08:49:20.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:b6378398:::76:head 2023-11-02T08:49:20.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:02bd68de:::77:head 2023-11-02T08:49:20.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:cc795d2d:::78:head 2023-11-02T08:49:20.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:630d4fea:::79:head 2023-11-02T08:49:20.667 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:e0d29ef5:::80:head 2023-11-02T08:49:20.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:fd6f13d2:::81:head 2023-11-02T08:49:20.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:606461d5:::82:head 2023-11-02T08:49:20.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:eadbdc43:::83:head 2023-11-02T08:49:20.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:8761d0bb:::84:head 2023-11-02T08:49:20.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:9ef0186f:::85:head 2023-11-02T08:49:20.668 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:e0d41294:::86:head 2023-11-02T08:49:20.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:961de695:::87:head 2023-11-02T08:49:20.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:1423148f:::88:head 2023-11-02T08:49:20.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:633a8fa2:::89:head 2023-11-02T08:49:20.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:a8653809:::90:head 2023-11-02T08:49:20.669 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:3dac8b33:::91:head 2023-11-02T08:49:20.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:35aad435:::92:head 2023-11-02T08:49:20.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:f6dcc343:::93:head 2023-11-02T08:49:20.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:dbbdad87:::94:head 2023-11-02T08:49:20.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:1cb48ce0:::95:head 2023-11-02T08:49:20.670 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:03cd461c:::96:head 2023-11-02T08:49:20.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:17a4ea99:::97:head 2023-11-02T08:49:20.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:9993c9a7:::98:head 2023-11-02T08:49:20.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:6394211c:::99:head 2023-11-02T08:49:20.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:94c7ae57:::100:head 2023-11-02T08:49:20.671 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:6fdee5bb:::101:head 2023-11-02T08:49:20.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:9a477fd1:::102:head 2023-11-02T08:49:20.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:eb850916:::103:head 2023-11-02T08:49:20.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:affc56b9:::104:head 2023-11-02T08:49:20.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:b42dc814:::105:head 2023-11-02T08:49:20.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:f319f8f0:::106:head 2023-11-02T08:49:20.672 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:9a40b9de:::107:head 2023-11-02T08:49:20.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:8b524f28:::108:head 2023-11-02T08:49:20.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:e3de589f:::109:head 2023-11-02T08:49:20.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:90f90a5b:::110:head 2023-11-02T08:49:20.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:a7b4f1d7:::111:head 2023-11-02T08:49:20.673 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:af51766e:::112:head 2023-11-02T08:49:20.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:b6f90bd1:::113:head 2023-11-02T08:49:20.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:e0261208:::114:head 2023-11-02T08:49:20.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c9569ef7:::115:head 2023-11-02T08:49:20.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:61bebe50:::116:head 2023-11-02T08:49:20.674 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:fe93412b:::117:head 2023-11-02T08:49:20.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d3d38bee:::118:head 2023-11-02T08:49:20.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:3100ba0c:::119:head 2023-11-02T08:49:20.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d0560ada:::120:head 2023-11-02T08:49:20.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:f0ea8b35:::121:head 2023-11-02T08:49:20.675 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:766f231a:::122:head 2023-11-02T08:49:20.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:a07a2582:::123:head 2023-11-02T08:49:20.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:bd7c6b3a:::124:head 2023-11-02T08:49:20.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:fb2ddaff:::125:head 2023-11-02T08:49:20.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:4408e1fe:::126:head 2023-11-02T08:49:20.676 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:ee1df7a7:::127:head 2023-11-02T08:49:20.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c3002909:::128:head 2023-11-02T08:49:20.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:4f48ffa9:::129:head 2023-11-02T08:49:20.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:edf38733:::130:head 2023-11-02T08:49:20.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c08425c0:::131:head 2023-11-02T08:49:20.677 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:5f902d98:::132:head 2023-11-02T08:49:20.678 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:41ea2c93:::133:head 2023-11-02T08:49:20.678 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:813cee13:::134:head 2023-11-02T08:49:20.678 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:0131818d:::135:head 2023-11-02T08:49:20.678 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:26ba5a85:::136:head 2023-11-02T08:49:20.679 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:381b8a5a:::137:head 2023-11-02T08:49:20.679 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:28797e47:::138:head 2023-11-02T08:49:20.679 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:bfca7f22:::139:head 2023-11-02T08:49:20.679 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:36807075:::140:head 2023-11-02T08:49:20.679 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:80b03975:::141:head 2023-11-02T08:49:20.680 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:5c15709b:::142:head 2023-11-02T08:49:20.680 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:f39ea15e:::143:head 2023-11-02T08:49:20.680 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:ea992956:::144:head 2023-11-02T08:49:20.680 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:48887b1c:::145:head 2023-11-02T08:49:20.680 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:9f24a9dd:::146:head 2023-11-02T08:49:20.681 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:987f100b:::147:head 2023-11-02T08:49:20.681 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d2dd3581:::148:head 2023-11-02T08:49:20.681 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:7fed1808:::149:head 2023-11-02T08:49:20.681 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c80b70e9:::150:head 2023-11-02T08:49:20.681 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:85ed90f9:::151:head 2023-11-02T08:49:20.681 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:36428b24:::152:head 2023-11-02T08:49:20.682 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d044c34a:::153:head 2023-11-02T08:49:20.682 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:7c18bf58:::154:head 2023-11-02T08:49:20.682 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d1c21232:::155:head 2023-11-02T08:49:20.682 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:a7a3c575:::156:head 2023-11-02T08:49:20.682 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:87da0633:::157:head 2023-11-02T08:49:20.683 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d5ac3822:::158:head 2023-11-02T08:49:20.683 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:3f20522d:::159:head 2023-11-02T08:49:20.683 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:6ca26563:::160:head 2023-11-02T08:49:20.683 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:532ce135:::161:head 2023-11-02T08:49:20.683 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:c78863e6:::162:head 2023-11-02T08:49:29.878 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:10.768 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:29.883 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:10.774 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:29.915 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:10.806 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:29.947 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:10.822 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:29.952 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:10.827 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:29.957 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:10.848 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:29.969 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:10.859 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:29.985 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:10.861 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:30.010 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:10.900 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:34.773 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:15.664 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:34.781 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:15.672 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:34.796 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:15.687 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:34.829 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:15.704 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:34.829 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:15.719 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:34.830 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:15.705 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:34.832 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:15.723 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:34.842 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:15.718 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:34.849 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:15.738 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:49.935 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:30.825 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:49.936 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:30.826 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:49.936 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:30.827 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:49.943 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:30.829 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:49.950 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:30.826 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:49.955 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:30.830 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:49.957 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:30.848 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:49.957 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:30.848 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:49.973 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:30.848 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:54.851 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:35.741 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:54.899 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:35.789 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:54.916 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:35.807 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:54.947 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:35.838 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:54.952 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:35.842 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:54.955 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:35.831 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:54.957 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:35.833 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:54.967 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:35.857 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:49:54.971 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:35.846 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:09.904 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:50.794 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:09.952 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:50.828 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:09.955 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:50.844 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:09.955 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:50.845 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:09.956 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:50.831 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:09.957 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:50.847 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:09.971 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:50.847 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:09.977 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:50.867 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:09.978 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:50.868 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:14.949 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:55.839 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:14.951 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:55.841 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:14.965 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:14:55.855 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:15.002 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:55.878 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:15.004 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:55.880 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:15.015 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:14:55.890 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:15.027 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:55.918 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:15.030 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:55.920 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:15.046 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:14:55.936 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:24.564 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:24.560+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:25.590 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:25.584+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:26.546 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:26.543+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:27.562 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:27.558+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:28.570 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:28.566+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:29.581 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:29.577+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:29.988 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:10.878 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:29.989 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:10.880 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.007 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:10.897 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.044 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:10.935 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.046 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:10.937 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.055 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:10.930 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.056 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:10.932 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.073 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:10.963 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.074 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:10.950 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.634 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:30.629+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:30.740 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:11.615 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.740 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:11.616 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.748 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:11.624 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.777 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:11.667 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.778 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:11.669 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.789 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:11.680 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.826 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:11.717 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.827 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:11.718 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:30.835 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:11.725 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:31.634 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:31.630+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:32.595 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:32.591+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:33.627 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:33.623+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:34.616 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:34.612+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:35.583 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:35.579+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:36.619 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:36.616+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:37.657 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:37.653+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:38.668 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:38.664+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:39.714 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:39.710+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:40.759 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:40.754+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:41.752 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:41.748+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:42.753 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:42.749+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:43.770 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:43.766+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:44.780 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:44.777+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:45.033 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:25.923 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:45.036 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:25.927 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:45.055 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:25.946 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:45.086 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:25.977 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:45.090 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:25.981 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:45.094 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:25.969 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:45.095 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:25.971 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:45.108 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:25.998 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:45.115 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:25.990 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:45.793 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:45.789+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:46.822 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:46.818+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:47.800 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:47.795+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:48.839 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:48.835+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:49.805 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:49.801+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:50.074 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:30.964 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:50.076 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:30.967 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:50.087 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:30.977 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:50.125 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:31.016 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:50.129 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:31.020 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:50.137 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:31.013 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:50.138 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:31.014 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:50.143 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:31.033 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:50.150 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:31.026 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:50:50.800 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:50.795+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:51.781 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:51.777+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:52.749 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:52.745+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:53.806 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:53.801+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:54.815 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:54.811+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:55.780 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:55.776+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:56.810 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:56.806+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:57.809 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:57.804+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:58.802 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:58.798+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:50:59.830 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:50:59.826+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:00.070 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:40.961 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:00.122 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:41.012 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:00.137 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:41.027 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:00.173 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:41.063 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:00.176 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:41.067 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:00.182 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:41.058 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:00.184 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:41.060 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:00.197 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:41.088 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:00.200 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:41.075 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:00.803 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:00.799+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:01.787 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:01.782+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:02.757 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:02.753+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:03.774 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:03.771+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:04.742 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:04.739+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:05.762 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:05.758+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:06.739 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:06.736+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:07.734 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:07.730+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:08.783 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:08.779+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:09.758 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:09.754+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:10.325 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:51.216 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:10.336 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:51.227 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:10.350 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:51.240 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:10.382 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:51.258 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:10.383 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:51.259 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:10.388 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:51.279 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:10.390 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:51.280 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:10.398 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:51.274 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:10.404 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:51.294 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:10.769 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:10.765+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:11.769 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:11.765+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:12.787 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:12.783+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:13.759 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:13.756+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:13.835 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:51:13.826+0000 12a62640 -1 osd.7 532 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T08:49:51.734451+0000 front 2023-11-02T08:49:51.719115+0000 (oldest deadline 2023-11-02T08:51:13.417456+0000) 2023-11-02T08:51:13.909 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:51:13.905+0000 12a62640 -1 osd.4 532 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T08:49:50.202574+0000 front 2023-11-02T08:49:50.215912+0000 (oldest deadline 2023-11-02T08:51:13.097594+0000) 2023-11-02T08:51:14.191 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:51:14.182+0000 12a62640 -1 osd.0 532 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T08:49:48.114566+0000 front 2023-11-02T08:49:48.111279+0000 (oldest deadline 2023-11-02T08:51:13.407302+0000) 2023-11-02T08:51:14.781 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:51:14.777+0000 12a62640 -1 osd.1 532 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:5316 77.c 77:365103e8:test-rados-api-smithi151-91630-48::foo0:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531) 2023-11-02T08:51:15.213 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:56.103 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:15.271 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:56.147 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:15.274 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:56.150 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:15.278 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:56.169 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:15.282 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:56.172 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:15.290 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:56.180 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:15.294 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:15:56.170 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:15.308 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:15:56.199 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:15.362 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:15:56.253 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:17.815 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:51:17.810+0000 14265640 -1 osd.2 535 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:51:20.583 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:01.473 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:20.587 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:01.477 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:20.616 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:01.507 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:20.637 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:01.512 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:20.641 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:01.517 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:20.662 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:01.537 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:20.666 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:01.556 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:20.671 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:01.562 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:20.697 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:01.587 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:30.253 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:11.144 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:30.257 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:11.148 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:30.280 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:11.171 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:30.306 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:11.182 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:30.309 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:11.185 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:30.310 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:11.200 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:30.316 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:11.206 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:30.330 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:11.205 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:30.337 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:11.228 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:31.119 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:12.009 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:31.119 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:12.010 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:31.120 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:12.011 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:31.120 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:12.011 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:31.129 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:12.019 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:31.129 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:12.020 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:31.148 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:12.023 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:31.148 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:12.024 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:31.159 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:12.035 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:50.329 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:31.218 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:50.333 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:31.223 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:50.360 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:31.251 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:50.385 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:31.275 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:50.388 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:31.264 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:50.391 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:31.266 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:50.406 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:31.296 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:50.415 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:31.291 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:50.434 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:31.324 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.105 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:31.980 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.109 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:31.982 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.115 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:31.990 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.174 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:32.065 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.175 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:32.066 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.191 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:32.082 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.202 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:32.087 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.202 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:32.089 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.207 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:32.098 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:51:51.927 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:7462ddf6:::163:head 2023-11-02T08:51:51.927 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:5d165639:::164:head 2023-11-02T08:51:51.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:f43765fc:::165:head 2023-11-02T08:51:51.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:b4c720e9:::166:head 2023-11-02T08:51:51.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:e694b040:::167:head 2023-11-02T08:51:51.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:afa38db2:::168:head 2023-11-02T08:51:51.928 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:77ba9f53:::169:head 2023-11-02T08:51:51.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:87495034:::170:head 2023-11-02T08:51:51.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:7c96bf0e:::171:head 2023-11-02T08:51:51.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:dbe346cc:::172:head 2023-11-02T08:51:51.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:e943ec24:::173:head 2023-11-02T08:51:51.929 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:f97a9c0c:::174:head 2023-11-02T08:51:51.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:6f26e74d:::175:head 2023-11-02T08:51:51.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:4f95e106:::176:head 2023-11-02T08:51:51.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:0e6f2f8f:::177:head 2023-11-02T08:51:51.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:05db05f1:::178:head 2023-11-02T08:51:51.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:38a78d66:::179:head 2023-11-02T08:51:51.930 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:d095610b:::180:head 2023-11-02T08:51:51.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:a1a9d709:::181:head 2023-11-02T08:51:51.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:1e5d39db:::182:head 2023-11-02T08:51:51.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:f7df4fb9:::183:head 2023-11-02T08:51:51.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:03a7f161:::184:head 2023-11-02T08:51:51.931 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:ba70721e:::185:head 2023-11-02T08:51:51.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:28e5662d:::186:head 2023-11-02T08:51:51.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:973d52de:::187:head 2023-11-02T08:51:51.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:4303eb1c:::188:head 2023-11-02T08:51:51.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:b990b48e:::189:head 2023-11-02T08:51:51.932 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:29b8165b:::190:head 2023-11-02T08:51:51.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:3547f197:::191:head 2023-11-02T08:51:51.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:7e260936:::192:head 2023-11-02T08:51:51.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:1abec7b1:::193:head 2023-11-02T08:51:51.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:10fdda93:::194:head 2023-11-02T08:51:51.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:15817eea:::195:head 2023-11-02T08:51:51.933 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:770bab57:::196:head 2023-11-02T08:51:51.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:ed9e13e7:::197:head 2023-11-02T08:51:51.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:71471a8f:::198:head 2023-11-02T08:51:51.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: checking for 201:10fb1d02:::199:head 2023-11-02T08:51:51.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21013 ms) 2023-11-02T08:51:51.934 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-11-02T08:51:51.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,0 2023-11-02T08:51:51.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: first is 1698914977 2023-11-02T08:51:51.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,0 2023-11-02T08:51:51.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,0 2023-11-02T08:51:51.935 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,1698914979,1698914980,0 2023-11-02T08:51:51.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,1698914979,1698914980,0 2023-11-02T08:51:51.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,1698914979,1698914980,0 2023-11-02T08:51:51.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,1698914979,1698914980,1698914982,1698914983,0 2023-11-02T08:51:51.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,1698914979,1698914980,1698914982,1698914983,0 2023-11-02T08:51:51.936 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,1698914979,1698914980,1698914982,1698914983,0 2023-11-02T08:51:51.937 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,1698914979,1698914980,1698914982,1698914983,1698914985,1698914986,0 2023-11-02T08:51:51.937 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,1698914979,1698914980,1698914982,1698914983,1698914985,1698914986,0 2023-11-02T08:51:51.937 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914977,1698914979,1698914980,1698914982,1698914983,1698914985,1698914986,0 2023-11-02T08:51:51.937 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: got ls 1698914980,1698914982,1698914983,1698914985,1698914986,1698914988,1698914989,0 2023-11-02T08:51:51.937 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: first now 1698914980, trimmed 2023-11-02T08:51:51.937 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22092 ms) 2023-11-02T08:51:51.938 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-11-02T08:51:51.938 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: foo0 2023-11-02T08:51:51.938 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-11-02T08:51:51.938 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (99070 ms) 2023-11-02T08:51:51.938 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-11-02T08:52:05.524 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:46.399 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:05.525 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:46.415 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:05.525 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:46.401 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:05.530 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:46.419 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:05.556 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:46.447 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:05.558 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:46.434 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:05.580 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:46.469 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:05.580 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:46.471 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:05.618 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:46.509 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:10.432 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:51.323 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:10.437 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:51.328 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:10.450 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:16:51.340 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:10.485 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:51.374 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:10.485 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:51.361 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:10.488 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:51.364 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:10.501 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:16:51.377 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:10.502 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:51.393 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:10.515 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:16:51.405 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:20.587 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:01.478 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:20.591 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:01.482 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:20.617 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:01.508 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:20.658 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:01.549 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:20.671 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:01.557 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:20.672 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:01.547 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:20.679 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:01.554 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:20.690 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:01.581 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:20.703 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:01.578 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:30.512 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:11.402 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:30.517 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:11.408 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:30.540 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:11.430 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:30.563 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:11.439 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:30.567 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:11.457 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:30.567 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:11.441 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:30.569 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:11.460 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:30.589 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:11.463 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:30.621 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:11.511 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:35.684 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:16.575 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:35.751 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:16.642 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:35.767 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:16.658 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:35.805 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:16.680 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:35.809 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:16.685 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:35.819 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:16.709 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:35.827 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:16.717 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:35.828 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:16.704 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:35.848 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:16.737 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:50.645 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:31.535 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:50.698 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:31.588 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:50.725 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:31.615 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:50.744 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:31.620 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:50.745 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:31.621 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:50.766 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:31.657 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:50.768 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:31.658 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:50.770 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:31.646 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:52:50.793 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:31.684 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:10.672 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:51.562 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:10.675 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:51.566 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:10.694 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:51.585 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:10.718 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:51.594 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:10.722 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:51.612 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:10.723 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:51.599 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:10.725 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:51.615 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:10.744 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:51.620 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:10.745 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:51.636 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:11.405 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:52.280 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:11.406 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:52.282 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:11.413 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:17:52.289 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:11.461 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:52.351 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:11.462 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:52.352 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:11.471 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:17:52.362 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:11.474 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:52.364 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:11.475 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:52.366 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:11.483 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:17:52.374 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:25.725 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:06.615 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:25.728 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:06.618 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:25.754 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:06.645 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:25.778 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:06.653 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:25.781 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:06.657 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:25.782 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:06.673 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:25.785 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:06.676 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:25.804 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:06.680 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:25.811 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:06.702 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:30.689 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:11.580 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:30.736 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:11.627 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:30.749 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:11.640 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:30.783 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:11.659 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:30.788 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:11.664 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:30.788 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:11.679 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:30.793 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:11.681 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:30.801 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:11.675 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:30.801 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:11.692 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:36.500 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:17.390 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:36.503 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:17.393 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:36.519 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:17.408 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:36.553 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:17.429 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:36.554 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:17.430 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:36.568 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:17.444 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:36.579 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:17.469 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:36.581 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:17.472 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:36.596 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:17.487 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:46.033 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:26.924 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:46.035 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:26.926 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:46.057 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:26.948 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:46.090 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:26.966 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:46.093 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:26.982 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:46.093 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:26.969 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:46.094 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:26.985 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:46.114 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:26.989 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:46.124 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:27.014 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:50.821 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:31.712 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:50.825 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:31.715 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:50.842 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:31.733 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:50.874 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:31.749 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:50.875 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:31.766 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:50.876 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:31.752 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:50.877 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:31.768 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:50.891 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:31.766 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:53:50.892 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:31.783 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:00.846 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:41.736 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:00.851 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:41.742 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:00.869 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:41.756 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:00.897 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:41.773 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:00.899 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:41.775 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:00.914 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:41.789 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:00.920 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:41.810 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:00.925 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:41.815 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:00.939 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:41.830 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:10.829 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:51.719 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:10.878 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:51.768 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:10.893 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:18:51.783 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:10.926 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:51.801 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:10.928 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:51.819 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:10.929 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:51.805 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:10.931 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:51.822 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:10.942 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:18:51.817 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:10.944 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:18:51.835 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:20.996 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:01.887 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:20.997 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:01.888 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:21.015 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:01.905 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:21.042 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:01.917 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:21.042 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:01.918 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:21.046 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:01.937 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:21.047 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:01.938 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:21.059 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:01.935 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:21.065 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:01.956 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:31.073 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:11.963 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:31.078 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:11.969 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:31.093 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:11.984 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:31.128 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:12.019 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:31.130 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:12.006 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:31.130 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:12.021 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:31.133 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:12.008 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:31.145 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:12.036 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:31.147 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:12.023 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:35.157 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:54:35.141+0000 2628f640 -1 snap_mapper.add_oid found existing snaps mapped on 77:7d0faeeb:test-rados-api-smithi151-91630-76::foo:28, removing 2023-11-02T08:54:35.165 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:54:35.156+0000 2faa2640 -1 snap_mapper.add_oid found existing snaps mapped on 77:7d0faeeb:test-rados-api-smithi151-91630-76::foo:28, removing 2023-11-02T08:54:40.972 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:54:40.955+0000 2628f640 -1 snap_mapper.add_oid found existing snaps mapped on 77:6d1b0259:test-rados-api-smithi151-91630-78::foo:29, removing 2023-11-02T08:54:40.986 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:54:40.977+0000 2f6a6640 -1 snap_mapper.add_oid found existing snaps mapped on 77:6d1b0259:test-rados-api-smithi151-91630-78::foo:29, removing 2023-11-02T08:54:40.992 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:21.883 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:40.995 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:21.886 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:41.017 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:21.907 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:41.047 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:21.937 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:41.047 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:21.923 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:41.050 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:21.926 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:41.050 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:21.941 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:41.072 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:21.947 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:41.081 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:21.970 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:51.148 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:32.039 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:51.150 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:32.041 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:51.168 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:32.059 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:51.195 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:32.071 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:51.196 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:32.072 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:51.199 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:32.090 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:51.200 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:32.091 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:51.214 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:32.089 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:51.220 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:32.111 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:56.041 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:36.932 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:56.045 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:36.936 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:56.057 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:36.947 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:56.094 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:36.969 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:56.095 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:36.984 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:56.097 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:36.973 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:56.097 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:36.987 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:56.106 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:36.997 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:54:56.107 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:36.983 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:07.767 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T08:55:07.752+0000 2628f640 -1 snap_mapper.add_oid found existing snaps mapped on 219:2cf0815c:test-rados-api-smithi151-91630-80::foo:2, removing 2023-11-02T08:55:07.777 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:55:07.766+0000 2aa98640 -1 snap_mapper.add_oid found existing snaps mapped on 219:2cf0815c:test-rados-api-smithi151-91630-80::foo:2, removing 2023-11-02T08:55:11.367 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:52.258 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:11.370 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:52.261 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:11.393 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:52.283 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:11.436 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:52.311 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:11.439 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:52.315 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:11.447 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:52.338 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:11.456 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:52.347 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:11.459 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:52.335 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:11.488 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:52.378 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:16.113 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:57.004 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:16.117 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:57.006 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:16.134 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:19:57.024 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:16.163 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:57.039 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:16.165 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:57.041 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:16.169 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:57.059 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:16.171 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:57.062 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:16.179 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:19:57.055 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:16.187 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:19:57.077 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:31.323 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:12.214 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:31.325 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:12.215 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:31.351 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:12.242 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:31.373 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:12.249 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:31.374 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:12.250 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:31.394 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:12.284 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:31.395 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:12.286 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:31.405 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:12.279 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:31.429 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:12.320 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:46.290 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:27.179 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:46.292 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:27.183 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:46.311 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:27.201 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:46.338 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:27.229 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:46.339 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:27.214 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:46.343 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:27.219 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:46.343 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:27.234 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:46.362 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:27.237 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:46.365 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:27.256 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:51.305 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:32.195 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:51.308 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:32.199 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:51.322 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:32.212 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:51.356 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:32.232 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:51.359 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:32.235 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:51.370 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:32.245 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:51.375 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:32.266 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:51.378 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:32.268 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:51.387 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:32.277 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:58.789 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:55:58.780+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:55:59.078 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:55:59.073+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:55:59.103 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:55:59.094+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:55:59.652 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:40.543 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:59.655 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:40.546 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:59.672 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:40.562 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:59.713 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:40.589 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:59.714 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:40.604 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:59.715 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:40.590 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:59.715 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:40.606 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:59.729 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:40.605 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:59.740 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:40.628 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:55:59.829 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:55:59.825+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:00.056 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:00.052+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:00.114 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:00.107+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:00.781 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:00.777+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:01.044 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:01.040+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:01.105 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:01.102+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:01.733 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:01.729+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:02.022 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:02.019+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:02.106 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:02.101+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:02.730 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:02.726+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:03.071 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:03.066+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:03.104 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:03.099+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:03.772 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:03.768+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:04.029 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:04.025+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:04.066 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:04.062+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:04.811 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:04.805+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:05.002 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:04.999+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:05.026 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:05.023+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:05.851 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:05.846+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:06.003 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:05.999+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:06.012 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:06.008+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:06.347 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:47.237 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:06.350 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:47.241 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:06.367 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:47.257 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:06.398 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:47.273 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:06.399 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:47.274 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:06.403 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:47.293 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:06.405 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:47.296 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:06.411 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:47.286 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:06.418 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:47.309 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:06.818 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:06.814+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:07.012 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:07.008+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:07.053 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:07.049+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:07.834 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:07.830+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:08.009 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:08.004+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:08.038 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:08.033+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:08.815 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:08.811+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:08.973 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:08.969+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:09.052 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:09.048+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:09.824 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:09.821+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:09.969 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:09.965+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:10.043 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:10.039+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:10.821 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:10.817+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:11.005 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:11.002+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:11.055 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:11.051+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:11.358 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:52.249 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:11.362 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:52.253 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:11.379 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:20:52.270 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:11.407 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:52.283 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:11.411 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:52.287 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:11.416 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:52.307 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:11.418 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:52.309 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:11.422 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:20:52.298 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:11.433 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:20:52.321 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:11.824 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:11.820+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:11.990 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:11.986+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:12.041 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:12.037+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:12.839 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:12.835+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:13.038 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:13.035+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:13.078 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:13.074+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:13.877 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:13.874+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:14.081 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:14.077+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:14.114 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:14.110+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:14.917 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:14.913+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:15.041 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:15.037+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:15.084 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:15.080+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:15.898 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:15.890+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:16.077 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:16.073+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:16.106 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:16.102+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:16.935 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:16.930+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:17.071 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:17.067+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:17.114 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:17.109+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:17.939 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:17.935+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:18.118 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:18.114+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:18.147 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:18.143+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:18.923 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:18.919+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:19.107 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:19.102+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:19.124 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:19.120+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:19.952 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:19.948+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:20.137 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:20.133+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:20.148 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:20.145+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:20.940 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:20.938+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:21.093 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:21.090+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:21.129 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:21.125+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:21.397 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:02.288 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:21.398 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:02.289 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:21.415 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:02.305 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:21.440 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:02.316 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:21.443 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:02.319 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:21.445 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:02.335 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:21.448 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:02.338 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:21.458 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:02.334 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:21.463 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:02.354 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:21.926 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:21.922+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:22.108 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:22.104+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:22.154 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:22.150+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:22.938 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:22.935+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:23.098 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:23.095+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:23.154 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:23.151+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:23.950 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:23.947+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:24.059 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:24.053+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:24.114 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:24.110+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:24.961 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:24.958+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:25.053 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:25.048+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:25.080 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:25.075+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:26.006 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:26.002+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:26.038 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:26.034+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:26.087 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:26.084+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:26.995 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:26.991+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:27.048 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:27.044+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:27.074 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:27.070+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:28.012 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:28.008+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:28.035 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:28.031+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:28.047 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:28.043+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:29.026 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:29.021+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:29.034 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:29.030+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:29.071 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:29.067+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:30.015 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:30.011+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:30.021 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:30.017+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:30.072 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:30.068+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:30.972 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:30.968+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:31.009 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:31.005+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:31.047 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:31.043+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:31.927 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:31.923+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:32.006 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:32.002+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:32.043 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:32.039+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:32.937 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:32.933+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:33.042 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:33.039+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:33.074 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:33.069+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:33.892 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:33.888+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:34.047 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:34.043+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:34.060 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:34.052+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:34.850 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:34.846+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:35.047 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:35.044+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:35.071 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:35.066+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:35.816 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:35.812+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:36.068 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:36.064+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:36.077 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:36.073+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:36.434 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:17.325 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:36.440 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:17.329 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:36.459 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:17.349 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:36.487 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:17.362 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:36.489 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:17.364 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:36.492 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:17.382 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:36.493 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:17.384 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:36.506 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:17.382 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:36.514 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:17.404 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:36.802 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:36.796+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:37.032 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:37.027+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:37.105 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:37.101+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:37.214 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:18.090 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:37.217 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:18.093 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:37.226 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:18.102 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:37.253 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:18.141 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:37.253 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:18.142 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:37.261 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:18.150 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:37.282 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:18.172 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:37.284 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:18.174 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:37.291 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:18.182 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:37.805 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:37.801+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:38.067 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:38.063+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:38.070 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:38.066+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:38.831 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:38.827+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:39.038 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:39.034+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:39.101 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:39.097+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:39.823 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:39.819+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:40.060 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:40.056+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:40.071 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:40.067+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:40.848 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:40.845+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:41.101 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:41.095+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:41.105 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:41.102+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:41.630 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:56:41.627+0000 12a62640 -1 osd.0 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:15.685802+0000 front 2023-11-02T08:55:15.686729+0000 (oldest deadline 2023-11-02T08:56:41.576342+0000) 2023-11-02T08:56:41.833 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:41.829+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:42.062 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:42.057+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:42.122 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:42.118+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:42.667 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:56:42.662+0000 12a62640 -1 osd.0 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:15.685802+0000 front 2023-11-02T08:55:15.686729+0000 (oldest deadline 2023-11-02T08:56:41.576342+0000) 2023-11-02T08:56:42.856 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:42.852+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:43.032 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:43.029+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:43.088 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:43.085+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:43.446 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:56:43.441+0000 12a62640 -1 osd.1 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:17.084786+0000 front 2023-11-02T08:55:17.086699+0000 (oldest deadline 2023-11-02T08:56:42.976258+0000) 2023-11-02T08:56:43.691 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:56:43.688+0000 12a62640 -1 osd.0 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:15.685802+0000 front 2023-11-02T08:55:15.686729+0000 (oldest deadline 2023-11-02T08:56:41.576342+0000) 2023-11-02T08:56:43.811 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:43.807+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:43.883 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:56:43.879+0000 12a62640 -1 osd.6 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:18.608782+0000 front 2023-11-02T08:55:18.600025+0000 (oldest deadline 2023-11-02T08:56:43.295151+0000) 2023-11-02T08:56:44.011 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:44.008+0000 12a62640 -1 osd.3 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:20.216729+0000 front 2023-11-02T08:55:20.212539+0000 (oldest deadline 2023-11-02T08:56:43.107436+0000) 2023-11-02T08:56:44.012 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:44.008+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:44.098 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:44.094+0000 12a62640 -1 osd.2 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:44.691 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:56:44.687+0000 12a62640 -1 osd.0 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:15.685802+0000 front 2023-11-02T08:55:15.686729+0000 (oldest deadline 2023-11-02T08:56:41.576342+0000) 2023-11-02T08:56:44.852 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T08:56:44.844+0000 12a62640 -1 osd.6 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:18.608782+0000 front 2023-11-02T08:55:18.600025+0000 (oldest deadline 2023-11-02T08:56:43.295151+0000) 2023-11-02T08:56:44.854 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:44.846+0000 12a62640 -1 osd.5 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:20.578567+0000 front 2023-11-02T08:55:20.577261+0000 (oldest deadline 2023-11-02T08:56:44.689346+0000) 2023-11-02T08:56:44.855 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T08:56:44.850+0000 12a62640 -1 osd.5 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:144 77.5 77:a71cea84:test-rados-api-smithi151-91630-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=1334b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:45.002 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:44.999+0000 12a62640 -1 osd.3 670 heartbeat_check: no reply from 172.21.15.178:6814 osd.4 since back 2023-11-02T08:55:20.216729+0000 front 2023-11-02T08:55:20.212539+0000 (oldest deadline 2023-11-02T08:56:43.107436+0000) 2023-11-02T08:56:45.004 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T08:56:45.000+0000 12a62640 -1 osd.3 670 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.2.537:147 77.0 77:01553b24:test-rados-api-smithi151-91630-84::1f3cd112d0661c8013c2564bab81c8c26a66839e:head [call cas.chunk_create_or_get_ref in=1661b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:45.063 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T08:56:45.060+0000 12a62640 -1 osd.2 671 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:7914 221.1 221:82cd7d94:test-rados-api-smithi151-91630-84::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e669) 2023-11-02T08:56:51.607 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:32.498 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:51.610 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:32.501 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:51.638 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:32.529 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:51.656 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:32.531 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:51.659 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:32.534 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:51.668 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:32.553 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:51.668 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:32.555 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:51.681 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:32.557 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:51.689 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:32.580 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:56.508 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:37.398 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:56.510 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:37.401 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:56.527 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:37.418 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:56.559 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:37.435 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:56.561 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:37.436 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:56.573 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:37.463 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:56.575 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:37.451 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:56.576 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:37.466 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:56:56.592 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:37.482 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:05.935 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:05.931+0000 14265640 -1 osd.4 683 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T08:57:06.708 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:47.583 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:06.709 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:47.584 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:06.715 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:47.605 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:06.719 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:47.610 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:06.728 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:47.604 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:06.740 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:47.630 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:06.771 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:47.662 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:06.772 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:47.663 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:06.788 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:47.679 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:16.584 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:57.475 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:16.587 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:57.477 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:16.602 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:21:57.493 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:16.635 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:57.511 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:16.636 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:57.512 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:16.651 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:21:57.527 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:16.659 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:57.550 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:16.661 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:57.552 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:16.683 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:21:57.574 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:21.597 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:02.487 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:21.599 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:02.490 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:21.615 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:02.505 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:21.644 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:02.519 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:21.646 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:02.537 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:21.647 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:02.522 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:21.650 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:02.540 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:21.657 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:02.533 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:21.666 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:02.556 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:26.898 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:26.890+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:27.870 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:27.865+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:28.919 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:28.915+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:29.968 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:29.963+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:31.003 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:30.998+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:32.011 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:32.007+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:33.003 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:32.998+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:34.020 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:34.016+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:35.048 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:35.044+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:36.057 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:36.053+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:36.612 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:17.503 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:36.665 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:17.556 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:36.689 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:17.579 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:36.708 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:17.584 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:36.711 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:17.587 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:36.717 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:17.607 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:36.719 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:17.610 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:36.727 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:17.602 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:36.735 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:17.626 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:37.081 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:37.077+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:37.372 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:18.248 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:37.373 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:18.249 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:37.381 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:18.257 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:37.417 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:18.307 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:37.418 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:18.309 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:37.433 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:18.324 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:37.435 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:18.326 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:37.436 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:18.327 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:37.448 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:18.338 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:38.107 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:38.102+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:39.138 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:39.134+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:40.169 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:40.164+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:40.812 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:40.808+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:41.218 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:41.214+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:41.855 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:41.852+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:42.236 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:42.232+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:42.864 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:42.860+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:43.283 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:43.279+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:43.820 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:43.817+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:44.324 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:44.320+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:44.838 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:44.832+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:45.288 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:45.283+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:45.861 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:45.857+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:46.281 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:46.276+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:46.857 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:46.853+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:47.282 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:47.278+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:47.826 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:47.823+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:48.264 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:48.260+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:48.783 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:48.779+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:49.247 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:49.243+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:49.759 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:49.755+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:50.214 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:50.210+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:50.799 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:50.794+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:51.253 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:51.249+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:51.823 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:51.819+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:52.243 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:52.239+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:52.833 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:52.829+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:53.199 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:53.195+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:53.881 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:53.877+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:54.219 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:54.215+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:54.846 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:54.843+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:55.191 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:55.187+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:55.845 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:55.841+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:56.209 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:56.206+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:56.742 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:37.632 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:56.747 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:37.637 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:56.770 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:37.661 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:56.793 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:37.669 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:56.794 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:37.671 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:56.796 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:37.687 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:56.801 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:37.692 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:56.814 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:37.690 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:56.822 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:37.712 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:56.889 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:56.885+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:57.228 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:57.223+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:57.525 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:38.401 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:57.526 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:38.402 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:57.534 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:38.410 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:57.573 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:38.464 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:57.575 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:38.465 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:57.586 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:38.476 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:57.636 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:38.527 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:57.637 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:38.528 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:57.647 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:38.536 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:57:57.914 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:57.911+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:58.204 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:58.201+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:58.931 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:58.927+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:57:59.186 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:57:59.179+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:57:59.970 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:57:59.966+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:00.228 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:58:00.224+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:58:00.932 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:58:00.929+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:01.213 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:58:01.209+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:58:01.940 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:58:01.936+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:02.176 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:58:02.172+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:58:02.948 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:58:02.944+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:03.222 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:58:03.218+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:58:03.935 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:58:03.932+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:04.186 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:58:04.183+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:58:04.923 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:58:04.919+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:05.162 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:58:05.158+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:58:05.940 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:58:05.936+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:06.154 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:58:06.149+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:58:06.977 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:58:06.974+0000 12a62640 -1 osd.4 686 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:07.167 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:58:07.164+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.13:92 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e682) 2023-11-02T08:58:07.974 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:58:07.969+0000 12a62640 -1 osd.4 687 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17258.0:8028 222.3 222:c638cae1:test-rados-api-smithi151-91630-86::foo:2 [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:08.170 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T08:58:08.167+0000 12a62640 -1 osd.0 686 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.4.684:75 77.12 77:4a1e8283:test-rados-api-smithi151-91630-86::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=819b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e686) 2023-11-02T08:58:11.767 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:52.657 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:11.773 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:52.663 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:11.808 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:52.699 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:11.821 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:52.696 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:11.822 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:52.698 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:11.843 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:52.734 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:11.845 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:52.721 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:11.845 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:52.735 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:11.868 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:52.759 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:12.376 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21278 ms) 2023-11-02T08:58:12.377 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-11-02T08:58:12.377 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23603 ms) 2023-11-02T08:58:12.377 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-11-02T08:58:12.377 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4345 ms) 2023-11-02T08:58:12.377 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-11-02T08:58:12.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4318 ms) 2023-11-02T08:58:12.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-11-02T08:58:12.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4326 ms) 2023-11-02T08:58:12.378 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-11-02T08:58:12.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5711 ms) 2023-11-02T08:58:12.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-11-02T08:58:12.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.379 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4546 ms) 2023-11-02T08:58:12.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-11-02T08:58:12.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40790 ms) 2023-11-02T08:58:12.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-11-02T08:58:12.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.380 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17875 ms) 2023-11-02T08:58:12.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-11-02T08:58:12.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5715 ms) 2023-11-02T08:58:12.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-11-02T08:58:12.381 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4846 ms) 2023-11-02T08:58:12.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-11-02T08:58:12.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26437 ms) 2023-11-02T08:58:12.382 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-11-02T08:58:12.383 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.383 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15980 ms) 2023-11-02T08:58:12.383 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-11-02T08:58:12.383 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.383 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6580 ms) 2023-11-02T08:58:12.383 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-11-02T08:58:12.384 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.384 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5493 ms) 2023-11-02T08:58:12.384 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-11-02T08:58:12.384 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.384 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: waiting for scrubs... 2023-11-02T08:58:12.385 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: done waiting 2023-11-02T08:58:12.385 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26041 ms) 2023-11-02T08:58:12.385 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-11-02T08:58:12.385 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.385 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11500 ms) 2023-11-02T08:58:12.386 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-11-02T08:58:12.386 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.386 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (87814 ms) 2023-11-02T08:58:12.386 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-11-02T08:58:12.386 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: require_osd_release = reef 2023-11-02T08:58:12.386 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (82295 ms) 2023-11-02T08:58:12.387 INFO:tasks.workunit.client.0.smithi151.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-11-02T08:58:16.862 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:57.753 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:16.915 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:57.806 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:16.931 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:22:57.822 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:16.959 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:57.834 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:16.962 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:57.837 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:16.967 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:57.858 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:16.970 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:57.861 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:16.977 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:22:57.853 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:16.990 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:22:57.881 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:21.836 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:02.726 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:21.841 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:02.731 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:21.849 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:02.725 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:21.853 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:02.728 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:21.854 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:02.744 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:21.860 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:02.751 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:21.930 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:02.806 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:21.957 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:02.847 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:21.966 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:02.855 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:26.939 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:07.830 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:26.944 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:07.834 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:26.960 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:07.851 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:26.994 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:07.870 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:26.995 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:07.871 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:26.998 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:07.889 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:27.001 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:07.892 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:27.010 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:07.886 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:27.019 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:07.910 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:37.100 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:17.990 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:37.111 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:18.002 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:37.139 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:18.029 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:37.167 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:18.058 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:37.177 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:18.053 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:37.179 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:18.055 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:37.182 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:18.073 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:37.213 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:18.089 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:37.234 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:18.124 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:41.865 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:22.756 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:41.913 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:22.804 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:41.929 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:22.820 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:41.966 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:22.842 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:41.968 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:22.859 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:41.969 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:22.845 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:41.971 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:22.862 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:41.982 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:22.858 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:41.984 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:22.875 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:57.074 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:37.962 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:57.122 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:38.013 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:57.142 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:38.032 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:57.171 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:38.047 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:57.177 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:38.052 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:57.177 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:38.068 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:57.180 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:38.071 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:57.197 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:38.073 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:58:57.201 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:38.091 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:02.146 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:43.035 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:02.205 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:43.095 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:02.234 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:43.125 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:02.267 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:43.142 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:02.268 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:43.144 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:02.276 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:43.167 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:02.279 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:43.170 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:02.291 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:43.166 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:02.306 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:43.196 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:03.118 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:43.994 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:03.119 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:43.995 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:03.129 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:44.005 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:03.215 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:44.105 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:03.216 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:44.107 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:03.225 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:44.116 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:03.248 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:44.139 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:03.249 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:44.140 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:03.259 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:44.149 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:15.559 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T08:59:15.529+0000 2628f640 -1 snap_mapper.add_oid found existing snaps mapped on 77:1b0be6da:test-rados-api-smithi151-91630-97::foo:31, removing 2023-11-02T08:59:15.576 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T08:59:15.560+0000 28a94640 -1 snap_mapper.add_oid found existing snaps mapped on 77:1b0be6da:test-rados-api-smithi151-91630-97::foo:31, removing 2023-11-02T08:59:17.414 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:58.304 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:17.416 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:58.306 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:17.436 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:23:58.327 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:17.464 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:58.355 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:17.470 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:58.361 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:17.473 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:58.349 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:17.476 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:58.352 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:17.498 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:23:58.389 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:17.502 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:23:58.378 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:32.274 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:13.164 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:32.274 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:13.165 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:32.308 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:13.198 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:32.320 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:13.196 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:32.322 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:13.198 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:32.329 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:13.219 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:32.332 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:13.223 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:32.351 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:13.227 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:32.362 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:13.253 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:37.424 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:18.315 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:37.435 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:18.319 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:37.449 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:18.340 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:37.472 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:18.348 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:37.478 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:18.351 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:37.490 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:18.366 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:37.497 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:18.388 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:37.502 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:18.391 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:37.517 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:18.407 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:47.352 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:28.242 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:47.353 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:28.228 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:47.358 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:28.249 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:47.373 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:28.248 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:47.394 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:28.270 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:47.409 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:28.300 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:47.411 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:28.302 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:47.441 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:28.332 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:47.451 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:28.342 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:57.333 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:38.224 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:57.386 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:38.276 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:57.428 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:38.318 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:57.438 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:38.314 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:57.438 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:38.329 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:57.439 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:38.330 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:57.442 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:38.318 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:57.459 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:38.350 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T08:59:57.462 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:38.338 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:02.511 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:43.402 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:02.515 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:43.406 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:02.520 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:43.396 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:02.524 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:43.399 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:02.537 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:43.427 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:02.540 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:43.415 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:02.578 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:43.468 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:02.579 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:43.470 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:02.596 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:43.486 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:17.466 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:58.357 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:17.473 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:58.363 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:17.519 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:58.409 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:17.530 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:58.420 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:17.532 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:58.408 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:17.533 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:58.409 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:17.534 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:58.424 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:17.568 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:58.443 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:17.584 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:58.475 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:18.318 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:59.194 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:18.319 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:59.195 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:18.329 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:24:59.205 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:18.388 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:59.279 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:18.389 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:59.280 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:18.403 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:24:59.294 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:18.407 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:59.298 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:18.408 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:59.299 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:18.418 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:24:59.309 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:23.804 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:04.694 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:23.812 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:04.703 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:23.837 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:04.727 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:23.858 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:04.733 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:23.859 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:04.735 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:23.868 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:04.759 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:23.870 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:04.760 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:23.876 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:04.752 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:23.888 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:04.778 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:32.510 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:13.400 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:32.514 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:13.405 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:32.530 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:13.420 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:32.563 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:13.439 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:32.565 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:13.456 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:32.566 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:13.442 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:32.569 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:13.460 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:32.578 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:13.454 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:32.585 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:13.476 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:37.629 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:18.519 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:37.635 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:18.526 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:37.648 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:18.539 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:37.681 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:18.571 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:37.684 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:18.560 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:37.686 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:18.577 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:37.690 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:18.566 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:37.699 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:18.590 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:37.700 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:18.576 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:49.821 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:49.817+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:50.775 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:50.771+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:51.817 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:51.813+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:52.647 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:33.537 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:52.651 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:33.542 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:52.672 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:33.563 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:52.697 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:33.573 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:52.700 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:33.576 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:52.700 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:33.591 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:52.704 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:33.595 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:52.717 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:33.593 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:52.721 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:33.612 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:52.794 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:52.790+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:53.772 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:53.768+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:54.758 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:54.755+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:55.741 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:55.736+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:56.746 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:56.743+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:57.728 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:38.618 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:57.734 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:38.624 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:57.747 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:38.637 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:57.751 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:57.748+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:57.778 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:38.653 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:57.797 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:38.673 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:57.801 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:38.692 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:57.802 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:38.693 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:57.809 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:38.684 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:57.815 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:38.705 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:00:58.728 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:58.726+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:00:59.723 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:00:59.720+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:00.695 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:00.692+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:01.656 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:01.653+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:02.657 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:02.653+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:03.648 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:03.644+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:04.683 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:04.678+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:05.677 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:05.673+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:06.704 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:06.699+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:07.699 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:48.589 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:07.702 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:48.593 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:07.722 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:25:48.613 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:07.734 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:07.730+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:07.756 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:48.647 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:07.758 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:48.633 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:07.760 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:48.636 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:07.760 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:48.651 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:07.775 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:25:48.651 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:07.778 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:25:48.669 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:08.745 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:08.742+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:09.706 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:09.702+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:10.707 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:10.703+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:11.672 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:11.668+0000 12a62640 -1 osd.0 784 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:12.739 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:12.735+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:13.747 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:13.744+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:14.734 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:14.730+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:15.724 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:15.720+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:16.775 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:16.771+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:17.786 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:17.782+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:18.808 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:18.804+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:19.824 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:19.820+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:20.826 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:20.821+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:21.803 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:21.799+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:22.752 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:03.643 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:22.755 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:03.646 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:22.776 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:03.667 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:22.787 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:22.783+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:22.801 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:03.677 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:22.804 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:03.681 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:22.808 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:03.699 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:22.811 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:03.702 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:22.825 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:03.701 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:22.833 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:03.724 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.405 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:04.280 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.405 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:04.281 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.415 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:04.291 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.458 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:04.349 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.493 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:04.383 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.501 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:04.392 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.511 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:04.402 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.513 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:04.403 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.524 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:04.414 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:23.768 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:23.763+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:24.781 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:24.777+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:25.803 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:25.799+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:26.788 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:26.784+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:27.836 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:27.832+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:28.829 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:28.825+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:29.798 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:29.794+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:30.788 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:30.783+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:31.762 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:31.758+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:32.721 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:32.717+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:33.724 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:33.721+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:33.962 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T09:01:33.957+0000 12a62640 -1 osd.3 786 heartbeat_check: no reply from 172.21.15.151:6804 osd.1 since back 2023-11-02T09:00:08.966949+0000 front 2023-11-02T09:00:08.965810+0000 (oldest deadline 2023-11-02T09:01:33.060494+0000) 2023-11-02T09:01:34.083 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T09:01:34.078+0000 12a62640 -1 osd.2 786 heartbeat_check: no reply from 172.21.15.151:6804 osd.1 since back 2023-11-02T09:00:07.662083+0000 front 2023-11-02T09:00:07.664064+0000 (oldest deadline 2023-11-02T09:01:33.545740+0000) 2023-11-02T09:01:34.761 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:34.756+0000 12a62640 -1 osd.0 786 heartbeat_check: no reply from 172.21.15.151:6804 osd.1 since back 2023-11-02T09:00:10.806573+0000 front 2023-11-02T09:00:10.808489+0000 (oldest deadline 2023-11-02T09:01:34.306461+0000) 2023-11-02T09:01:34.763 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:34.757+0000 12a62640 -1 osd.0 786 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:34.982 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T09:01:34.977+0000 12a62640 -1 osd.3 786 heartbeat_check: no reply from 172.21.15.151:6804 osd.1 since back 2023-11-02T09:00:08.966949+0000 front 2023-11-02T09:00:08.965810+0000 (oldest deadline 2023-11-02T09:01:33.060494+0000) 2023-11-02T09:01:35.096 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T09:01:35.092+0000 12a62640 -1 osd.2 786 heartbeat_check: no reply from 172.21.15.151:6804 osd.1 since back 2023-11-02T09:00:07.662083+0000 front 2023-11-02T09:00:07.664064+0000 (oldest deadline 2023-11-02T09:01:33.545740+0000) 2023-11-02T09:01:35.766 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:01:35.762+0000 12a62640 -1 osd.0 787 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22141.0:9 236.18 236.18 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e782) 2023-11-02T09:01:35.914 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T09:01:35.911+0000 12a62640 -1 osd.5 786 heartbeat_check: no reply from 172.21.15.151:6804 osd.1 since back 2023-11-02T09:00:12.096389+0000 front 2023-11-02T09:00:12.111584+0000 (oldest deadline 2023-11-02T09:01:35.587509+0000) 2023-11-02T09:01:37.944 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:18.834 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:37.946 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:18.837 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:37.971 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:18.861 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:37.998 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:18.874 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:38.000 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:18.891 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:38.001 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:18.892 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:38.021 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:18.897 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:38.022 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:18.912 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:38.042 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:18.918 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:42.815 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:23.705 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:42.819 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:23.710 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:42.836 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:23.726 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:42.866 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:23.741 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:42.868 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:23.744 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:42.872 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:23.763 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:42.874 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:23.765 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:42.884 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:23.759 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:42.889 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:23.779 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:52.961 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:33.852 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:52.964 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:33.854 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:52.987 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:33.878 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:53.011 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:33.887 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:53.014 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:33.905 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:53.016 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:33.891 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:53.016 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:33.906 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:53.039 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:33.915 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:01:53.043 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:33.933 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:00.084 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:00.080+0000 14265640 -1 osd.1 808 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-02T09:02:02.898 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:43.788 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:02.904 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:43.794 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:02.928 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:43.818 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:02.951 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:43.827 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:02.952 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:43.828 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:02.956 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:43.846 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:02.957 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:43.848 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:02.974 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:43.850 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:02.992 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:43.883 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:08.009 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:48.898 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:08.068 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:48.958 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:08.082 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:26:48.972 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:08.114 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:48.990 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:08.116 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:48.992 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:08.126 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:49.017 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:08.127 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:49.018 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:08.129 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:26:49.005 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:08.141 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:26:49.032 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:22.965 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:03.854 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:22.966 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:03.856 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:22.980 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:03.871 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:23.013 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:03.889 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:23.016 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:03.892 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:23.031 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:03.906 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:23.045 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:03.935 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:23.045 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:03.936 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:23.061 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:03.951 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:27.977 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:08.867 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:27.982 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:08.871 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:27.993 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:08.884 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:28.031 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:08.904 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:28.031 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:08.905 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:28.032 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:08.922 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:28.034 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:08.925 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:28.039 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:08.915 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:28.046 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:08.937 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:41.558 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:41.554+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:42.566 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:42.562+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:43.012 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:23.903 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:43.016 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:23.907 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:43.032 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:23.923 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:43.068 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:23.958 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:43.069 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:23.945 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:43.071 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:23.947 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:43.071 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:23.962 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:43.086 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:23.961 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:43.087 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:23.978 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:43.614 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:43.610+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:44.567 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:44.563+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:45.553 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:45.550+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:46.578 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:46.574+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:47.590 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:47.586+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:48.035 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:28.925 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:48.036 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:28.926 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:48.047 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:28.937 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:48.083 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:28.959 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:48.084 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:28.960 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:48.085 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:28.975 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:48.089 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:28.980 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:48.095 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:28.971 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:48.101 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:28.992 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:48.541 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:48.537+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:49.565 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:49.561+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:50.594 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:50.591+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:51.597 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:51.593+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:52.584 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:52.580+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:53.584 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:53.580+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:54.556 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:54.552+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:54.679 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:35.570 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:54.684 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:35.574 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:54.701 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:35.592 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:54.732 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:35.608 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:54.740 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:35.615 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:54.740 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:35.631 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:54.745 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:35.634 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:54.758 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:35.634 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:54.760 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:35.650 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:02:55.513 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:55.509+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:56.557 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:56.554+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:57.540 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:57.536+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:58.509 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:58.506+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:02:59.481 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:02:59.474+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:00.528 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:00.524+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:01.542 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:01.538+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:02.537 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:02.534+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:03.110 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:44.000 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.115 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:44.006 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.131 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:44.022 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.161 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:44.037 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.162 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:44.038 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.174 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:44.050 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.181 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:44.072 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.184 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:44.075 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.203 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:44.093 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.527 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:03.523+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:03.759 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:44.634 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.759 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:44.635 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.767 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:44.643 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.833 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:44.723 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.834 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:44.724 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.842 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:44.732 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.858 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:44.749 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.859 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:44.750 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:03.867 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:44.758 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:04.550 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:04.546+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:05.594 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:05.590+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:06.563 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:06.558+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:07.598 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:07.595+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:08.563 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:08.559+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:09.573 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:09.569+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:10.595 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:10.592+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:11.574 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:11.570+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:12.527 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:12.523+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:13.506 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:13.502+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:14.515 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:14.511+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:15.468 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:15.464+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:16.425 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:16.421+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:17.409 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:17.405+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:18.149 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:59.039 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:18.150 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:59.041 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:18.166 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:27:59.057 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:18.198 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:59.074 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:18.200 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:59.076 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:18.204 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:59.095 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:18.205 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:59.096 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:18.217 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:27:59.093 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:18.224 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:27:59.114 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:18.371 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:18.366+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:19.371 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:19.365+0000 12a62640 -1 osd.1 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 ever on either front or back, first ping sent 2023-11-02T09:01:59.030165+0000 (oldest deadline 2023-11-02T09:03:19.030165+0000) 2023-11-02T09:03:19.372 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:19.368+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:20.040 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T09:03:20.036+0000 12a62640 -1 osd.3 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T09:01:54.911605+0000 front 2023-11-02T09:01:54.918165+0000 (oldest deadline 2023-11-02T09:03:19.606717+0000) 2023-11-02T09:03:20.386 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:20.381+0000 12a62640 -1 osd.1 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 ever on either front or back, first ping sent 2023-11-02T09:01:59.030165+0000 (oldest deadline 2023-11-02T09:03:19.030165+0000) 2023-11-02T09:03:20.386 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:20.382+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:21.038 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T09:03:21.033+0000 12a62640 -1 osd.3 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T09:01:54.911605+0000 front 2023-11-02T09:01:54.918165+0000 (oldest deadline 2023-11-02T09:03:19.606717+0000) 2023-11-02T09:03:21.216 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T09:03:21.212+0000 12a62640 -1 osd.7 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T09:01:59.190953+0000 front 2023-11-02T09:01:59.214620+0000 (oldest deadline 2023-11-02T09:03:20.757134+0000) 2023-11-02T09:03:21.370 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:21.365+0000 12a62640 -1 osd.1 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 ever on either front or back, first ping sent 2023-11-02T09:01:59.030165+0000 (oldest deadline 2023-11-02T09:03:19.030165+0000) 2023-11-02T09:03:21.372 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:03:21.368+0000 12a62640 -1 osd.1 814 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.22147.0:108 237.3s0 237.2a631073 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e814) 2023-11-02T09:03:21.389 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T09:03:21.385+0000 12a62640 -1 osd.5 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T09:01:57.129070+0000 front 2023-11-02T09:01:57.131583+0000 (oldest deadline 2023-11-02T09:03:20.620806+0000) 2023-11-02T09:03:22.036 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T09:03:22.032+0000 12a62640 -1 osd.3 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T09:01:54.911605+0000 front 2023-11-02T09:01:54.918165+0000 (oldest deadline 2023-11-02T09:03:19.606717+0000) 2023-11-02T09:03:22.132 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:03:22.128+0000 12a62640 -1 osd.0 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T09:01:58.360326+0000 front 2023-11-02T09:01:58.385609+0000 (oldest deadline 2023-11-02T09:03:21.178474+0000) 2023-11-02T09:03:22.180 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T09:03:22.175+0000 12a62640 -1 osd.7 814 heartbeat_check: no reply from 172.21.15.151:6820 osd.2 since back 2023-11-02T09:01:59.190953+0000 front 2023-11-02T09:01:59.214620+0000 (oldest deadline 2023-11-02T09:03:20.757134+0000) 2023-11-02T09:03:23.183 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:04.073 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:23.190 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:04.078 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:23.203 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:04.093 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:23.245 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:04.136 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:23.248 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:04.139 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:23.253 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:04.129 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:23.257 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:04.133 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:23.270 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:04.160 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:23.271 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:04.146 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:33.202 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:14.092 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:33.206 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:14.097 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:33.229 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:14.119 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:33.262 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:14.137 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:33.262 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:14.153 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:33.263 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:14.139 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:33.266 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:14.156 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:33.279 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:14.155 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:33.283 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:14.174 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:43.243 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:24.134 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:43.247 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:24.138 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:43.264 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:24.154 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:43.298 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:24.173 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:43.298 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:24.189 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:43.302 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:24.192 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:43.302 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:24.175 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:43.315 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:24.205 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:43.315 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:24.188 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:53.268 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:34.158 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:53.272 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:34.162 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:53.287 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:34.178 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:53.321 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:34.211 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:53.322 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:34.197 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:53.323 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:34.199 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:53.324 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:34.214 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:53.335 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:34.211 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:03:53.336 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:34.226 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:03.301 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:44.191 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:03.303 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:44.193 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:03.316 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:44.206 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:03.349 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:44.224 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:03.349 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:44.239 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:03.352 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:44.228 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:03.354 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:44.245 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:03.364 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:44.240 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:03.371 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:44.262 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:13.323 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:54.214 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:13.329 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:54.220 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:13.344 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:28:54.234 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:13.373 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:54.264 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:13.376 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:54.252 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:13.377 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:54.253 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:13.379 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:54.269 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:13.391 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:28:54.267 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:13.392 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:28:54.282 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:23.353 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:04.243 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:23.355 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:04.245 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:23.369 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:04.259 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:23.404 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:04.279 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:23.405 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:04.281 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:23.418 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:04.294 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:23.424 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:04.315 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:23.430 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:04.320 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:23.443 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:04.334 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:33.383 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:14.274 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:33.386 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:14.277 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:33.404 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:14.294 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:33.437 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:14.313 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:33.438 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:14.314 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:33.439 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:14.329 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:33.443 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:14.334 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:33.452 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:14.327 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:33.457 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:14.348 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:43.414 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:24.304 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:43.417 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:24.308 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:43.434 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:24.324 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:43.466 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:24.356 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:43.467 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:24.342 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:43.469 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:24.345 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:43.474 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:24.365 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:43.482 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:24.358 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:43.493 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:24.384 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:53.446 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:34.336 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:53.448 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:34.339 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:53.464 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:34.354 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:53.497 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:34.373 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:53.500 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:34.376 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:53.513 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:34.389 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:53.519 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:34.410 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:53.522 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:34.413 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:04:53.535 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:34.426 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:03.471 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:44.361 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:03.475 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:44.366 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:03.497 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:44.388 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:03.525 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:44.401 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:03.528 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:44.404 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:03.538 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:44.428 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:03.541 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:44.417 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:03.542 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:44.433 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:03.558 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:44.449 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:13.534 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:54.424 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:13.537 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:54.428 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:13.557 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:29:54.447 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:13.582 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:54.457 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:13.583 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:54.459 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:13.585 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:54.476 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:13.589 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:54.480 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:13.597 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:29:54.472 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:13.604 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:29:54.495 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:23.490 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:04.380 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:23.540 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:04.430 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:23.540 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:04.430 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:23.547 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:04.437 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:23.564 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:04.455 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:23.565 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:04.455 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:23.596 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:04.472 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:23.597 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:04.473 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:23.609 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:04.485 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:33.568 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:14.458 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:33.570 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:14.461 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:33.587 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:14.477 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:33.619 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:14.494 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:33.620 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:14.510 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:33.621 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:14.497 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:33.622 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:14.513 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:33.635 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:14.511 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:33.638 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:14.529 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:43.591 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:24.481 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:43.595 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:24.486 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:43.614 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:24.505 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:43.641 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:24.517 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:43.645 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:24.521 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:43.658 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:24.534 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:43.661 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:24.552 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:43.665 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:24.556 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:43.678 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:24.569 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:47.293 INFO:tasks.ceph.osd.2.smithi151.stderr:*** Caught signal (Aborted) ** 2023-11-02T09:05:47.294 INFO:tasks.ceph.osd.2.smithi151.stderr: in thread 2faa2640 thread_name:tp_osd_tp 2023-11-02T09:05:47.351 INFO:tasks.ceph.osd.2.smithi151.stderr: ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable) 2023-11-02T09:05:47.352 INFO:tasks.ceph.osd.2.smithi151.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2023-11-02T09:05:47.352 INFO:tasks.ceph.osd.2.smithi151.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2023-11-02T09:05:47.352 INFO:tasks.ceph.osd.2.smithi151.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x226) [0xb9a846] 2023-11-02T09:05:47.352 INFO:tasks.ceph.osd.2.smithi151.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x70) [0xb9a940] 2023-11-02T09:05:47.352 INFO:tasks.ceph.osd.2.smithi151.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x248) [0xbaaec8] 2023-11-02T09:05:47.353 INFO:tasks.ceph.osd.2.smithi151.stderr: 6: ceph-osd(+0xaa3444) [0xbab444] 2023-11-02T09:05:47.353 INFO:tasks.ceph.osd.2.smithi151.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2023-11-02T09:05:47.353 INFO:tasks.ceph.osd.2.smithi151.stderr: 8: clone() 2023-11-02T09:05:47.417 INFO:tasks.ceph.osd.2.smithi151.stderr:2023-11-02T09:05:47.411+0000 2faa2640 -1 *** Caught signal (Aborted) ** 2023-11-02T09:05:47.417 INFO:tasks.ceph.osd.2.smithi151.stderr: in thread 2faa2640 thread_name:tp_osd_tp 2023-11-02T09:05:47.417 INFO:tasks.ceph.osd.2.smithi151.stderr: 2023-11-02T09:05:47.417 INFO:tasks.ceph.osd.2.smithi151.stderr: ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable) 2023-11-02T09:05:47.418 INFO:tasks.ceph.osd.2.smithi151.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2023-11-02T09:05:47.418 INFO:tasks.ceph.osd.2.smithi151.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2023-11-02T09:05:47.418 INFO:tasks.ceph.osd.2.smithi151.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x226) [0xb9a846] 2023-11-02T09:05:47.418 INFO:tasks.ceph.osd.2.smithi151.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x70) [0xb9a940] 2023-11-02T09:05:47.418 INFO:tasks.ceph.osd.2.smithi151.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x248) [0xbaaec8] 2023-11-02T09:05:47.419 INFO:tasks.ceph.osd.2.smithi151.stderr: 6: ceph-osd(+0xaa3444) [0xbab444] 2023-11-02T09:05:47.419 INFO:tasks.ceph.osd.2.smithi151.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2023-11-02T09:05:47.419 INFO:tasks.ceph.osd.2.smithi151.stderr: 8: clone() 2023-11-02T09:05:47.419 INFO:tasks.ceph.osd.2.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-02T09:05:47.419 INFO:tasks.ceph.osd.2.smithi151.stderr: 2023-11-02T09:05:47.877 INFO:tasks.ceph.osd.2.smithi151.stderr: 0> 2023-11-02T09:05:47.411+0000 2faa2640 -1 *** Caught signal (Aborted) ** 2023-11-02T09:05:47.877 INFO:tasks.ceph.osd.2.smithi151.stderr: in thread 2faa2640 thread_name:tp_osd_tp 2023-11-02T09:05:47.878 INFO:tasks.ceph.osd.2.smithi151.stderr: 2023-11-02T09:05:47.878 INFO:tasks.ceph.osd.2.smithi151.stderr: ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable) 2023-11-02T09:05:47.878 INFO:tasks.ceph.osd.2.smithi151.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2023-11-02T09:05:47.878 INFO:tasks.ceph.osd.2.smithi151.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2023-11-02T09:05:47.879 INFO:tasks.ceph.osd.2.smithi151.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x226) [0xb9a846] 2023-11-02T09:05:47.879 INFO:tasks.ceph.osd.2.smithi151.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x70) [0xb9a940] 2023-11-02T09:05:47.879 INFO:tasks.ceph.osd.2.smithi151.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x248) [0xbaaec8] 2023-11-02T09:05:47.879 INFO:tasks.ceph.osd.2.smithi151.stderr: 6: ceph-osd(+0xaa3444) [0xbab444] 2023-11-02T09:05:47.879 INFO:tasks.ceph.osd.2.smithi151.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2023-11-02T09:05:47.880 INFO:tasks.ceph.osd.2.smithi151.stderr: 8: clone() 2023-11-02T09:05:47.880 INFO:tasks.ceph.osd.2.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-02T09:05:47.880 INFO:tasks.ceph.osd.2.smithi151.stderr: 2023-11-02T09:05:48.263 INFO:tasks.ceph.osd.2.smithi151.stderr: -437> 2023-11-02T09:05:47.411+0000 2faa2640 -1 *** Caught signal (Aborted) ** 2023-11-02T09:05:48.263 INFO:tasks.ceph.osd.2.smithi151.stderr: in thread 2faa2640 thread_name:tp_osd_tp 2023-11-02T09:05:48.263 INFO:tasks.ceph.osd.2.smithi151.stderr: 2023-11-02T09:05:48.264 INFO:tasks.ceph.osd.2.smithi151.stderr: ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable) 2023-11-02T09:05:48.264 INFO:tasks.ceph.osd.2.smithi151.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2023-11-02T09:05:48.264 INFO:tasks.ceph.osd.2.smithi151.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2023-11-02T09:05:48.264 INFO:tasks.ceph.osd.2.smithi151.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x226) [0xb9a846] 2023-11-02T09:05:48.265 INFO:tasks.ceph.osd.2.smithi151.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x70) [0xb9a940] 2023-11-02T09:05:48.265 INFO:tasks.ceph.osd.2.smithi151.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x248) [0xbaaec8] 2023-11-02T09:05:48.265 INFO:tasks.ceph.osd.2.smithi151.stderr: 6: ceph-osd(+0xaa3444) [0xbab444] 2023-11-02T09:05:48.265 INFO:tasks.ceph.osd.2.smithi151.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2023-11-02T09:05:48.265 INFO:tasks.ceph.osd.2.smithi151.stderr: 8: clone() 2023-11-02T09:05:48.266 INFO:tasks.ceph.osd.2.smithi151.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-02T09:05:48.266 INFO:tasks.ceph.osd.2.smithi151.stderr: 2023-11-02T09:05:50.889 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:30:12.610 86100== 2023-11-02T09:05:50.890 INFO:tasks.ceph.osd.2.smithi151.stderr:==00:00:30:12.610 86100== Exit program on first error (--exit-on-first-error=yes) 2023-11-02T09:05:50.974 INFO:tasks.ceph.osd.2.smithi151.stderr:daemon-helper: command failed with exit status 42 2023-11-02T09:05:51.726 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:32.617 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:51.728 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:32.619 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:51.757 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:32.648 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:51.796 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:32.671 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:51.800 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:32.676 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:51.823 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:32.699 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:51.917 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:32.808 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:51.918 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:32.809 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:51.938 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:32.829 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:57.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-02T09:05:58.696 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:39.587 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:58.700 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:39.591 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:58.713 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:39.603 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:58.744 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:39.635 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:58.747 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:39.638 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:58.749 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:39.625 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:58.750 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:39.626 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:58.759 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:39.650 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:05:58.761 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:39.637 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:03.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~6s 2023-11-02T09:06:03.793 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:44.683 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:03.794 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:44.685 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:03.805 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:44.695 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:03.842 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:44.717 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:03.842 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:44.718 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:03.846 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:44.737 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:03.847 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:44.738 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:03.852 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:44.728 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:03.858 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:44.749 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:09.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~12s 2023-11-02T09:06:15.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~19s 2023-11-02T09:06:18.758 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:59.648 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:18.759 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:59.650 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:18.776 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:30:59.666 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:18.805 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:59.696 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:18.807 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:59.683 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:18.808 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:59.699 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:18.811 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:59.687 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:18.828 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:30:59.704 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:18.831 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:30:59.721 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:22.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~25s 2023-11-02T09:06:23.810 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:04.701 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:23.858 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:04.734 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:23.858 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:04.749 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:23.860 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:04.736 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:23.873 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:04.749 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:23.876 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:04.767 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:23.908 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:04.799 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:23.911 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:04.802 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:23.925 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:04.816 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:28.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~31s 2023-11-02T09:06:34.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~37s 2023-11-02T09:06:38.888 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:19.778 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:38.891 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:19.782 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:38.911 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:19.802 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:38.944 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:19.834 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:38.944 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:19.820 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:38.945 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:19.822 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:38.946 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:19.837 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:38.962 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:19.838 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:38.963 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:19.853 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:40.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~43s 2023-11-02T09:06:43.906 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:24.796 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:43.908 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:24.799 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:43.923 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:24.814 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:43.951 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:24.827 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:43.955 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:24.845 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:43.955 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:24.831 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:43.958 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:24.848 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:43.966 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:24.842 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:43.973 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:24.864 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:46.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2023-11-02T09:06:53.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~56s 2023-11-02T09:06:58.940 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:39.831 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:58.946 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:39.837 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:58.971 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:39.862 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:58.995 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:39.871 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:58.996 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:39.872 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:59.004 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:39.895 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:59.007 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:39.898 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:59.015 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:39.891 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:59.030 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:39.921 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:06:59.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~62s 2023-11-02T09:07:03.953 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:44.843 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:03.957 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:44.848 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:03.973 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:44.864 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:04.007 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:44.883 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:04.008 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:44.899 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:04.011 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:44.902 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:04.012 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:44.888 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:04.025 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:44.901 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:04.027 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:44.918 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:05.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~68s 2023-11-02T09:07:11.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~75s 2023-11-02T09:07:18.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~81s 2023-11-02T09:07:18.999 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:59.889 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:19.003 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:59.894 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:19.023 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:31:59.913 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:19.056 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:59.947 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:19.059 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:59.950 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:19.060 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:59.936 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:19.063 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:59.939 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:19.080 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:31:59.971 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:19.081 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:31:59.956 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.009 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:04.899 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.011 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:04.901 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.024 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:04.914 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.057 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:04.933 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.064 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:04.955 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.065 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:04.956 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.071 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:04.947 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.076 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:04.967 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.086 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:04.962 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:24.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~87s 2023-11-02T09:07:30.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~93s 2023-11-02T09:07:36.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~99s 2023-11-02T09:07:39.065 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:19.956 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:39.069 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:19.959 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:39.089 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:19.980 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:39.121 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:19.997 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:39.122 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:20.013 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:39.123 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:19.999 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:39.124 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:20.015 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:39.143 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:20.019 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:39.146 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:20.036 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:42.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~106s 2023-11-02T09:07:44.087 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:24.978 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:44.092 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:24.982 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:44.107 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:24.998 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:44.137 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:25.012 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:44.139 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:25.015 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:44.142 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:25.033 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:44.144 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:25.035 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:44.151 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:25.027 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:44.157 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:25.048 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:49.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~112s 2023-11-02T09:07:55.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~118s 2023-11-02T09:07:59.120 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:40.011 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:59.126 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:40.016 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:59.144 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:40.034 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:59.175 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:40.066 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:59.177 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:40.053 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:59.181 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:40.072 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:59.182 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:40.057 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:59.201 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:40.077 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:07:59.202 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:40.093 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:01.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~124s 2023-11-02T09:08:04.130 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:45.021 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:04.135 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:45.025 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:04.148 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:32:45.038 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:04.182 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:45.058 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:04.185 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:45.075 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:04.189 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:45.079 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:04.189 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:45.065 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:04.203 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:32:45.079 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:04.204 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:32:45.095 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:07.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2023-11-02T09:08:13.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~137s 2023-11-02T09:08:19.179 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:00.069 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:19.182 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:00.072 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:19.198 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:00.089 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:19.230 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:00.121 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:19.231 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:00.106 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:19.233 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:00.109 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:19.234 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:00.125 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:19.250 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:00.126 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:19.252 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:00.143 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:20.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~143s 2023-11-02T09:08:24.193 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:05.083 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:24.195 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:05.086 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:24.213 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:05.104 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:24.246 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:05.122 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:24.251 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:05.127 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:24.262 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:05.138 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:24.272 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:05.162 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:24.277 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:05.167 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:24.288 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:05.178 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:26.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~149s 2023-11-02T09:08:32.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~155s 2023-11-02T09:08:38.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~161s 2023-11-02T09:08:39.243 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:20.133 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:39.246 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:20.137 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:39.266 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:20.157 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:39.303 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:20.179 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:39.304 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:20.180 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:39.310 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:20.201 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:39.314 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:20.205 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:39.322 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:20.198 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:39.334 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:20.225 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.258 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:25.149 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.261 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:25.152 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.263 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:25.139 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.267 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:25.143 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.276 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:25.167 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.280 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:25.156 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.314 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:25.204 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.315 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:25.205 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.330 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:25.220 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:44.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~168s 2023-11-02T09:08:51.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~174s 2023-11-02T09:08:57.393 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~180s 2023-11-02T09:08:59.309 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:40.199 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:59.309 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:40.200 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:59.326 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:40.216 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:59.361 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:40.237 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:59.363 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:40.239 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:59.366 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:40.257 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:59.369 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:40.260 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:59.382 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:40.258 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:08:59.385 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:40.276 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:03.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~186s 2023-11-02T09:09:03.853 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:44.743 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:03.856 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:44.746 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:03.878 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:44.768 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:03.909 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:44.785 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:03.910 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:44.786 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:03.924 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:44.815 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:03.925 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:44.816 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:03.926 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:44.802 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:03.948 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:44.839 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:04.484 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:45.360 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:04.485 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:45.361 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:04.493 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:33:45.368 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:04.528 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:45.418 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:04.528 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:45.419 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:04.541 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:33:45.431 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:04.563 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:45.453 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:04.564 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:45.455 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:04.574 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:33:45.465 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:09.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~193s 2023-11-02T09:09:16.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~199s 2023-11-02T09:09:19.379 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:00.270 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:19.384 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:00.274 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:19.403 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:00.294 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:19.436 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:00.311 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:19.438 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:00.329 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:19.443 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:00.319 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:19.445 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:00.336 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:19.463 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:00.339 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:19.467 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:00.357 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:22.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~205s 2023-11-02T09:09:24.396 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:05.286 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:24.398 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:05.288 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:24.409 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:05.300 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:24.449 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:05.340 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:24.450 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:05.326 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:24.451 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:05.341 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:24.453 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:05.329 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:24.465 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:05.356 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:24.468 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:05.343 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:28.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~211s 2023-11-02T09:09:34.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~217s 2023-11-02T09:09:39.439 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:20.330 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:39.441 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:20.332 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:39.463 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:20.353 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:39.497 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:20.373 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:39.500 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:20.376 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:39.503 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:20.393 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:39.506 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:20.396 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:39.518 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:20.393 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:39.521 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:20.412 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:40.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~224s 2023-11-02T09:09:44.449 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:25.340 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:44.455 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:25.345 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:44.467 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:25.358 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:44.500 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:25.390 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:44.504 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:25.395 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:44.505 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:25.381 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:44.508 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:25.384 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:44.519 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:25.410 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:44.522 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:25.398 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:09:47.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~230s 2023-11-02T09:09:53.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~236s 2023-11-02T09:09:59.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~242s 2023-11-02T09:10:04.507 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:45.397 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:04.510 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:45.401 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:04.531 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:34:45.421 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:04.561 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:45.452 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:04.562 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:45.438 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:04.565 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:45.456 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:04.566 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:45.442 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:04.589 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:34:45.464 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:04.597 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:34:45.484 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:05.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~248s 2023-11-02T09:10:11.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~255s 2023-11-02T09:10:18.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~261s 2023-11-02T09:10:24.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~267s 2023-11-02T09:10:24.560 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:05.451 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:24.566 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:05.456 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:24.583 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:05.474 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:24.612 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:05.488 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:24.615 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:05.491 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:24.633 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:05.508 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:24.634 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:05.525 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:24.638 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:05.529 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:24.656 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:05.547 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:29.530 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:10.420 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:29.578 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:10.468 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:29.582 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:10.457 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:29.591 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:10.466 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:29.592 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:10.482 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:29.601 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:10.477 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:29.646 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:10.537 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:29.649 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:10.540 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:29.661 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:10.552 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:30.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~273s 2023-11-02T09:10:36.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~279s 2023-11-02T09:10:42.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2023-11-02T09:10:44.615 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:25.506 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:44.619 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:25.510 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:44.638 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:25.528 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:44.668 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:25.544 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:44.672 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:25.563 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:44.674 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:25.565 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:44.675 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:25.550 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:44.691 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:25.582 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:44.692 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:25.567 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:49.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~292s 2023-11-02T09:10:49.704 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:30.594 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:49.735 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:30.626 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:49.747 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:30.637 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:49.747 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:30.623 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:49.761 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:30.652 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:49.778 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:30.654 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:49.782 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:30.673 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:49.789 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:30.664 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:49.792 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:30.683 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:55.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~298s 2023-11-02T09:10:59.662 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:40.553 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:59.666 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:40.557 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:59.684 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:40.574 85770== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:59.716 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:40.591 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:59.716 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:40.606 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:59.719 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:40.595 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:59.721 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:40.612 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:59.735 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:40.611 85497== Warning: unimplemented fcntl command: 1036 2023-11-02T09:10:59.737 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:40.627 85776== Warning: unimplemented fcntl command: 1036 2023-11-02T09:11:01.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~304s 2023-11-02T09:11:01.596 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-02T09:11:02.803 INFO:tasks.ceph.osd.0:Sent signal 15 2023-11-02T09:11:02.804 INFO:tasks.ceph.osd.1:Sent signal 15 2023-11-02T09:11:02.805 INFO:tasks.ceph.osd.3:Sent signal 15 2023-11-02T09:11:02.805 INFO:tasks.ceph.osd.4:Sent signal 15 2023-11-02T09:11:02.806 INFO:tasks.ceph.osd.5:Sent signal 15 2023-11-02T09:11:02.806 INFO:tasks.ceph.osd.6:Sent signal 15 2023-11-02T09:11:02.807 INFO:tasks.ceph.osd.7:Sent signal 15 2023-11-02T09:11:02.807 INFO:tasks.ceph.mon.a:Sent signal 15 2023-11-02T09:11:02.807 INFO:tasks.ceph.mon.c:Sent signal 15 2023-11-02T09:11:02.808 INFO:tasks.ceph.mon.b:Sent signal 15 2023-11-02T09:11:02.808 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-11-02T09:11:02.809 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-11-02T09:11:02.820 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T09:11:02.807+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 86111) UID: 0 2023-11-02T09:11:02.821 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T09:11:02.806+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86089) UID: 0 2023-11-02T09:11:02.821 INFO:tasks.ceph.osd.3.smithi151.stderr:2023-11-02T09:11:02.807+0000 b47e640 -1 osd.3 818 *** Got signal Terminated *** 2023-11-02T09:11:02.822 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:11:02.804+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86070) UID: 0 2023-11-02T09:11:02.822 INFO:tasks.ceph.osd.0.smithi151.stderr:2023-11-02T09:11:02.806+0000 b47e640 -1 osd.0 818 *** Got signal Terminated *** 2023-11-02T09:11:02.823 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:11:02.805+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86079) UID: 0 2023-11-02T09:11:02.823 INFO:tasks.ceph.osd.1.smithi151.stderr:2023-11-02T09:11:02.807+0000 b47e640 -1 osd.1 818 *** Got signal Terminated *** 2023-11-02T09:11:02.824 INFO:tasks.ceph.osd.4.smithi178.stderr:2023-11-02T09:11:02.808+0000 b47e640 -1 osd.4 818 *** Got signal Terminated *** 2023-11-02T09:11:02.824 INFO:tasks.ceph.mon.a.smithi151.stderr:2023-11-02T09:11:02.808+0000 9f34640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 85752) UID: 0 2023-11-02T09:11:02.828 INFO:tasks.ceph.mon.a.smithi151.stderr:2023-11-02T09:11:02.810+0000 9f34640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-02T09:11:02.829 INFO:tasks.ceph.mon.c.smithi151.stderr:2023-11-02T09:11:02.810+0000 9f34640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 85754) UID: 0 2023-11-02T09:11:02.833 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T09:11:02.820+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 86118) UID: 0 2023-11-02T09:11:02.837 INFO:tasks.ceph.mon.c.smithi151.stderr:2023-11-02T09:11:02.812+0000 9f34640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-11-02T09:11:02.839 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T09:11:02.821+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 86114) UID: 0 2023-11-02T09:11:02.839 INFO:tasks.ceph.osd.5.smithi178.stderr:2023-11-02T09:11:02.822+0000 b47e640 -1 osd.5 818 *** Got signal Terminated *** 2023-11-02T09:11:02.840 INFO:tasks.ceph.osd.7.smithi178.stderr:2023-11-02T09:11:02.822+0000 b47e640 -1 osd.7 818 *** Got signal Terminated *** 2023-11-02T09:11:02.840 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T09:11:02.821+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 86116) UID: 0 2023-11-02T09:11:02.841 INFO:tasks.ceph.osd.6.smithi178.stderr:2023-11-02T09:11:02.823+0000 b47e640 -1 osd.6 818 *** Got signal Terminated *** 2023-11-02T09:11:02.841 INFO:tasks.ceph.mon.b.smithi178.stderr:2023-11-02T09:11:02.824+0000 9f34640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 85490) UID: 0 2023-11-02T09:11:02.841 INFO:tasks.ceph.mon.b.smithi178.stderr:2023-11-02T09:11:02.826+0000 9f34640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-11-02T09:11:03.010 INFO:tasks.ceph.mgr.y.smithi151.stderr:daemon-helper: command crashed with signal 15 2023-11-02T09:11:03.383 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:44.258 85497== 2023-11-02T09:11:03.384 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:35:44.258 85497== Exit program on first error (--exit-on-first-error=yes) 2023-11-02T09:11:03.400 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:44.291 85770== 2023-11-02T09:11:03.401 INFO:tasks.ceph.mon.a.smithi151.stderr:==00:00:35:44.291 85770== Exit program on first error (--exit-on-first-error=yes) 2023-11-02T09:11:03.421 INFO:tasks.ceph.mon.b.smithi178.stderr:daemon-helper: command failed with exit status 42 2023-11-02T09:11:03.445 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:44.336 85776== 2023-11-02T09:11:03.446 INFO:tasks.ceph.mon.c.smithi151.stderr:==00:00:35:44.336 85776== Exit program on first error (--exit-on-first-error=yes) 2023-11-02T09:11:03.609 INFO:tasks.ceph.mon.a.smithi151.stderr:daemon-helper: command failed with exit status 42 2023-11-02T09:11:03.610 INFO:tasks.ceph.mon.c.smithi151.stderr:daemon-helper: command failed with exit status 42 2023-11-02T14:37:25.889 INFO:tasks.workunit.client.0.smithi151.stderr:++ cleanup 2023-11-02T14:37:25.915 INFO:tasks.workunit.client.0.smithi151.stderr:++ pkill -P 91284 2023-11-02T14:37:25.972 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-02T14:37:25.973 INFO:tasks.workunit.client.0.smithi151.stderr:++ true 2023-11-02T14:37:25.973 INFO:tasks.workunit.client.0.smithi151.stdout: a 2023-11-02T14:37:25.974 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-11-02T14:37:25.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-02T14:37:26.249 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi151 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2023-11-02T14:37:26.326 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=438a2aec82e9401d94b9f9a99c0d8599 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi151 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2023-11-02T14:37:26.329 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-02T14:37:26.341 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-02T14:37:26.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-02T14:37:26.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.455+0000 7f54e1a41640 10 monclient: get_monmap_and_config 2023-11-02T14:37:26.456 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.455+0000 7f54e1a41640 10 monclient: build_initial_monmap 2023-11-02T14:37:26.457 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.455+0000 7f54e1a41640 10 monclient: monmap: 2023-11-02T14:37:26.457 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2023-11-02T14:37:26.457 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 4d1a52ed-5f96-4808-8336-f35436fa00ba 2023-11-02T14:37:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2023-11-02T14:37:26.455817+0000 2023-11-02T14:37:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:created 2023-11-02T14:37:26.455817+0000 2023-11-02T14:37:26.458 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2023-11-02T14:37:26.459 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2023-11-02T14:37:26.459 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a-legacy 2023-11-02T14:37:26.459 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c-legacy 2023-11-02T14:37:26.459 INFO:teuthology.orchestra.run.smithi151.stderr:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.noname-b 2023-11-02T14:37:26.460 INFO:teuthology.orchestra.run.smithi151.stderr:3: v2:172.21.15.151:3300/0 mon.noname-a 2023-11-02T14:37:26.460 INFO:teuthology.orchestra.run.smithi151.stderr:4: v2:172.21.15.178:3300/0 mon.noname-c 2023-11-02T14:37:26.460 INFO:teuthology.orchestra.run.smithi151.stderr: 2023-11-02T14:37:26.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.456+0000 7f54e1a41640 1 Processor -- start 2023-11-02T14:37:26.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.456+0000 7f54e1a41640 1 -- start start 2023-11-02T14:37:26.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.456+0000 7f54e1a41640 10 monclient: init 2023-11-02T14:37:26.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 10 monclient: _reopen_session rank -1 2023-11-02T14:37:26.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-11-02T14:37:26.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54dc0a94c0 addr v1:172.21.15.151:6789/0 2023-11-02T14:37:26.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 0x7f54dc0aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:26.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 10 monclient(hunting): picked mon.noname-a con 0x7f54dc0a98a0 addr v2:172.21.15.151:3300/0 2023-11-02T14:37:26.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54dc13b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:26.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 10 monclient(hunting): picked mon.noname-c con 0x7f54dc141a60 addr v2:172.21.15.178:3300/0 2023-11-02T14:37:26.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54dc0a94c0 2023-11-02T14:37:26.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:26.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54dc0a98a0 2023-11-02T14:37:26.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:26.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54dc141a60 2023-11-02T14:37:26.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:26.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54e1a41640 10 monclient(hunting): authenticate will time out at 23017.811967s 2023-11-02T14:37:26.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 msgr2=0x7f54dc0aef70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:26.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54dc0accc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:26.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 0x7f54dc0aef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:26.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54dc13b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:26.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.457+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54dc13b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:26.658 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.658+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54dc0accc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:26.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.658+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 msgr2=0x7f54dc0aef70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:26.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.658+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 0x7f54dc0aef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:26.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.658+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54dc13b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:26.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:26.658+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54dc13b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:27.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.059+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54dc0accc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:27.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.059+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 msgr2=0x7f54dc0aef70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:27.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.059+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 0x7f54dc0aef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:27.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.059+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54dc13b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:27.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.059+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54dc13b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.860+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 msgr2=0x7f54dc0aef70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.860+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54dc0accc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.860+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 0x7f54dc0aef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:27.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.860+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54dc13b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:27.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:27.860+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54dc13b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:29.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:29.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:29.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:29.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:29.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54dc13b9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:29.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54dc13b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:29.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 msgr2=0x7f54dc0aef70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:29.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54dc0a98a0 0x7f54dc0aef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:29.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54dc0accc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:29.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 10 monclient: _add_conns ranks=[2,1] 2023-11-02T14:37:29.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 0x7f54bc007cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:29.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc0058a0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:37:29.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc008330 addr v1:172.21.15.178:6789/0 2023-11-02T14:37:29.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc008330 2023-11-02T14:37:29.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:29.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc0058a0 2023-11-02T14:37:29.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:29.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:29.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:29.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.457+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc008330 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:29.658 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.658+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:29.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.658+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:29.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:29.658+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc008330 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:30.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:30.058+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:30.058+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:30.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:30.058+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc008330 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:30.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:30.859+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:30.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:30.859+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:30.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:30.859+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc008330 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:32.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:32.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:32.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:32.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:32.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:32.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc0058a0 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:32.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc008330 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:32.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-11-02T14:37:32.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 0x7f54bc00c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:32.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00c0c0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:37:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:37:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54dc0a94c0 addr v1:172.21.15.151:6789/0 2023-11-02T14:37:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54dc141a60 addr v2:172.21.15.178:3300/0 2023-11-02T14:37:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54dc0a94c0 2023-11-02T14:37:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:32.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54dc141a60 2023-11-02T14:37:32.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:32.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00c0c0 2023-11-02T14:37:32.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54bc00f250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 msgr2=0x7f54bc00c500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 0x7f54bc00c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:32.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.456+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:32.658 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.657+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54bc00f250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:32.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.657+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 msgr2=0x7f54bc00c500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:32.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.657+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 0x7f54bc00c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:32.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.657+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:32.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:32.657+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:33.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.058+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54bc00f250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:33.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.058+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 msgr2=0x7f54bc00c500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:33.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.058+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 0x7f54bc00c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:33.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.058+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:33.076 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.058+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:33.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.859+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54bc00f250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:33.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.859+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 msgr2=0x7f54bc00c500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:33.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.859+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 0x7f54bc00c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:33.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.859+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:33.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:33.859+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:35.458 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:35.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:35.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:35.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:35.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 msgr2=0x7f54bc00c500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:35.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c0c0 0x7f54bc00c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:35.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:35.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:35.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54dc0a94c0 legacy=0x7f54bc00f250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:35.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,3] 2023-11-02T14:37:35.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc00af20 addr v1:172.21.15.178:6789/0 2023-11-02T14:37:35.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 0x7f54bc00b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:35.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00b360 addr v2:172.21.15.151:3300/0 2023-11-02T14:37:35.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc00af20 2023-11-02T14:37:35.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:35.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00b360 2023-11-02T14:37:35.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:35.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 msgr2=0x7f54bc00b770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:35.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.456+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 0x7f54bc00b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:35.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.457+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00af20 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:35.658 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.657+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 msgr2=0x7f54bc00b770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:35.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.657+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 0x7f54bc00b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:35.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:35.658+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00af20 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:36.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:36.058+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 msgr2=0x7f54bc00b770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:36.059 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:36.058+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 0x7f54bc00b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:36.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:36.059+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00af20 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:36.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:36.859+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 msgr2=0x7f54bc00b770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:36.860 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:36.859+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 0x7f54bc00b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:36.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:36.860+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00af20 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:38.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:38.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:38.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:38.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:38.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 msgr2=0x7f54bc00b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:38.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00b360 0x7f54bc00b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:38.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00af20 legacy=0x7f54bc00a7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:38.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-11-02T14:37:38.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:38.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:37:38.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54dc141a60 addr v2:172.21.15.178:3300/0 2023-11-02T14:37:38.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:38.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00c500 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:37:38.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc0087e0 addr v1:172.21.15.178:6789/0 2023-11-02T14:37:38.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc0087e0 2023-11-02T14:37:38.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:38.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54dc141a60 2023-11-02T14:37:38.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:38.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00c500 2023-11-02T14:37:38.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:38.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:38.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:38.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc0087e0 legacy=0x7f54bc00d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:38.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:38.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.457+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:38.659 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.658+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:38.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.658+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:38.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.658+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc0087e0 legacy=0x7f54bc00d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:38.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.658+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:38.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:38.658+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:39.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.059+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:39.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.059+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:39.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.059+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc0087e0 legacy=0x7f54bc00d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:39.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.059+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:39.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.059+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:39.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.860+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:39.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.860+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:39.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.860+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:39.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.860+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:39.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:39.860+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc0087e0 legacy=0x7f54bc00d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:41.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:41.459 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:41.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:41.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:41.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:41.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:41.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 msgr2=0x7f54bc007cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:41.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54dc141a60 0x7f54bc007cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:41.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc0087e0 legacy=0x7f54bc00d240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:41.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,0] 2023-11-02T14:37:41.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 0x7f54bc00edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:41.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00e960 addr v2:172.21.15.151:3300/0 2023-11-02T14:37:41.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00f310 addr v1:172.21.15.151:6789/0 2023-11-02T14:37:41.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc00f310 2023-11-02T14:37:41.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:41.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00e960 2023-11-02T14:37:41.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:41.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f310 legacy=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:41.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 msgr2=0x7f54bc00edd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:41.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.457+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 0x7f54bc00edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:41.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.658+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f310 legacy=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:41.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.658+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 msgr2=0x7f54bc00edd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:41.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:41.658+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 0x7f54bc00edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:42.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:42.059+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f310 legacy=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:42.060 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:42.059+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 msgr2=0x7f54bc00edd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:42.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:42.059+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 0x7f54bc00edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:42.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:42.860+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 msgr2=0x7f54bc00edd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:42.861 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:42.860+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f310 legacy=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:42.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:42.860+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 0x7f54bc00edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:44.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:44.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:44.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:44.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:44.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 msgr2=0x7f54bc00edd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:44.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00e960 0x7f54bc00edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:44.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f310 legacy=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:44.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-11-02T14:37:44.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:37:44.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00fe30 addr v1:172.21.15.151:6789/0 2023-11-02T14:37:44.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:44.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00c500 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:37:44.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:44.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00ed40 addr v2:172.21.15.151:3300/0 2023-11-02T14:37:44.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc00fe30 2023-11-02T14:37:44.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:44.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00ed40 2023-11-02T14:37:44.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:44.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00c500 2023-11-02T14:37:44.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fe30 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:44.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.457+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:44.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.458+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:44.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.458+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:44.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.458+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:44.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.458+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:44.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.659+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fe30 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:44.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.659+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:44.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.659+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:44.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.659+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:44.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:44.659+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.060+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.060+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fe30 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:45.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.060+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:45.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.060+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:45.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.060+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:45.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.861+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:45.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.861+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:45.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.861+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:45.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.861+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fe30 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:45.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:45.861+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:47.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:47.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:47.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:47.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:47.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:47.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:47.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:47.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:47.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fe30 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:47.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,4] 2023-11-02T14:37:47.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc00a7f0 addr v1:172.21.15.178:6789/0 2023-11-02T14:37:47.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:47.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc007cf0 addr v2:172.21.15.178:3300/0 2023-11-02T14:37:47.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc00a7f0 2023-11-02T14:37:47.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:47.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc007cf0 2023-11-02T14:37:47.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:47.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:47.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc00d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:47.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.458+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:47.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.659+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc00d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.659+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:47.659+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:48.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:48.060+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc00d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:48.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:48.060+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:48.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:48.060+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:48.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:48.861+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:48.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:48.861+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc00d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:48.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:48.861+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:50.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:50.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:50.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:50.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:50.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:50.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:50.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc00d200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:50.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-11-02T14:37:50.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:50.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00c500 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:37:50.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:50.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00ed40 addr v2:172.21.15.151:3300/0 2023-11-02T14:37:50.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:37:50.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:50.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc007cf0 addr v2:172.21.15.178:3300/0 2023-11-02T14:37:50.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:50.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00ed40 2023-11-02T14:37:50.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:50.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc007cf0 2023-11-02T14:37:50.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:50.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc00ac10 con 0x7f54bc00c500 2023-11-02T14:37:50.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:50.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:50.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.458+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:50.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.659+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:50.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.659+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:50.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.659+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:50.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.659+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:50.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.659+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:50.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:50.659+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:51.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.060+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:51.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.060+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:51.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.060+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:51.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.060+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:51.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.060+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:51.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.060+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:51.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.861+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:51.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.861+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:51.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.861+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:51.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.861+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:51.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.861+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:51.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:51.861+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:53.460 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:53.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:53.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:53.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:53.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:53.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c500 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:53.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:53.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:53.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:53.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00ed40 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:53.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,0] 2023-11-02T14:37:53.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc00a7f0 addr v1:172.21.15.178:6789/0 2023-11-02T14:37:53.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00fde0 addr v1:172.21.15.151:6789/0 2023-11-02T14:37:53.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc00fde0 2023-11-02T14:37:53.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc00a7f0 2023-11-02T14:37:53.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:53.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fde0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:53.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.458+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:53.660 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.659+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fde0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:53.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:53.659+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:54.061 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:54.060+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fde0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:54.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:54.060+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:54.862 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:54.861+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fde0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:54.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:54.861+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:56.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:56.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:56.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:56.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:56.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00a7f0 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:56.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00fde0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:56.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-11-02T14:37:56.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:56.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:37:56.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc007cf0 addr v2:172.21.15.178:3300/0 2023-11-02T14:37:56.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00ed40 addr v1:172.21.15.151:6789/0 2023-11-02T14:37:56.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:56.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00fde0 addr v2:172.21.15.151:3300/0 2023-11-02T14:37:56.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc00ed40 2023-11-02T14:37:56.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:56.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:37:56.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:56.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc007cf0 2023-11-02T14:37:56.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:56.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:56.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:56.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:56.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:56.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.459+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:56.661 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.660+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:56.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.660+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:56.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.660+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:56.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.660+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:56.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:56.660+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:57.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.061+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:57.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.061+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:57.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.061+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:57.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.061+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:57.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.061+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:37:57.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.862+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:37:57.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.862+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:37:57.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.862+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:57.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.862+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:37:57.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:57.862+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:37:59.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:37:59.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:37:59.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:37:59.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:37:59.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:59.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:59.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:59.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:37:59.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:37:59.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,2] 2023-11-02T14:37:59.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.178:6789/0 2023-11-02T14:37:59.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:37:59.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00a7f0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:37:59.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc00c4a0 2023-11-02T14:37:59.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:37:59.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00a7f0 2023-11-02T14:37:59.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:37:59.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:59.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:37:59.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.459+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:37:59.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.660+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:37:59.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.660+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:37:59.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:37:59.660+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:00.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:00.061+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:00.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:00.061+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:00.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:00.061+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:00.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:00.862+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:00.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:00.862+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:00.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:00.862+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:02.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:02.461 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:02.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:02.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:02.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:02.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:02.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:02.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-11-02T14:38:02.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:02.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:02.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00fde0 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:02.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00ed40 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:02.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:02.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc00ed40 2023-11-02T14:38:02.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc00c4a0 2023-11-02T14:38:02.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:02.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:38:02.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:02.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:02.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:02.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:02.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.459+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:02.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.660+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:02.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.660+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:02.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.660+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:02.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:02.660+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:03.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:03.061+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:03.062 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:03.061+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:03.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:03.061+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:03.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:03.061+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:03.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:03.862+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:03.863 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:03.862+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:03.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:03.862+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:03.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:03.862+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:05.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:05.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:05.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:05.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:05.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:05.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:05.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00e1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:05.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00ed40 legacy=0x7f54bc010250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:05.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 10 monclient: _add_conns ranks=[4,2] 2023-11-02T14:38:05.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:05.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc007cf0 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:05.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:05.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00a7f0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:05.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:05.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc007cf0 2023-11-02T14:38:05.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:05.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00a7f0 2023-11-02T14:38:05.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:05.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:05.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:05.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:05.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.460+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:05.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.661+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:05.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.661+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:05.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.661+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:05.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:05.661+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:06.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:06.062+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:06.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:06.062+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:06.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:06.062+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:06.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:06.062+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:06.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:06.863+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:06.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:06.863+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:06.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:06.863+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:06.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:06.863+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:08.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:08.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:08.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:08.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:08.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:08.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:08.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 msgr2=0x7f54bc012980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:08.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc007cf0 0x7f54bc012980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:08.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-11-02T14:38:08.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:08.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00fde0 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:08.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:08.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:08.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00a7f0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:08.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:08.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc00c4a0 2023-11-02T14:38:08.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:08.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:38:08.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:08.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:08.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:08.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00a7f0 2023-11-02T14:38:08.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:08.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:08.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:08.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.460+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:08.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.661+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:08.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.661+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:08.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.661+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:08.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.661+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:08.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:08.661+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:09.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.062+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:09.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.062+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:09.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.062+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:09.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.062+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:09.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.062+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:09.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.863+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:09.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.863+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:09.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.863+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:09.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.863+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:09.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:09.863+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:11.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:11.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:11.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:11.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:11.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:11.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:11.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:11.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:11.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:11.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 10 monclient: _add_conns ranks=[4,1] 2023-11-02T14:38:11.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 0x7f54bc015ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:11.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc00ed40 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:11.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc007cf0 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:11.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc007cf0 2023-11-02T14:38:11.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:11.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00ed40 2023-11-02T14:38:11.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:11.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc015ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:11.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc016420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:11.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.460+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 0x7f54bc015ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:11.662 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.661+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc016420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:11.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.661+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc015ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:11.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:11.661+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 0x7f54bc015ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:12.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:12.062+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc015ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:12.063 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:12.062+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc016420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:12.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:12.062+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 0x7f54bc015ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:12.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:12.863+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc016420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:12.864 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:12.863+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc015ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:12.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:12.863+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 0x7f54bc015ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:14.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:14.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:14.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:14.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:14.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 msgr2=0x7f54bc015ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:14.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00ed40 0x7f54bc015ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:14.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc016420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:14.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-11-02T14:38:14.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 0x7f54bc0136b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:14.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc0132d0 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:14.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:14.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:14.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:14.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00fde0 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:14.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc00c4a0 2023-11-02T14:38:14.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:14.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:38:14.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:14.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc0132d0 2023-11-02T14:38:14.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:14.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc016b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:14.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:14.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:14.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 msgr2=0x7f54bc0136b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:14.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.461+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 0x7f54bc0136b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:14.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.662+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc016b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:14.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.662+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:14.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.662+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:14.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.662+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 msgr2=0x7f54bc0136b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:14.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:14.662+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 0x7f54bc0136b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:15.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.063+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:15.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.063+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc016b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:15.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.063+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:15.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.063+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 msgr2=0x7f54bc0136b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:15.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.063+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 0x7f54bc0136b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:15.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.864+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:15.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.864+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc016b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:15.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.864+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:15.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.864+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 msgr2=0x7f54bc0136b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:15.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:15.864+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 0x7f54bc0136b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:17.462 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:17.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:17.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:17.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:17.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 msgr2=0x7f54bc0136b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:17.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0132d0 0x7f54bc0136b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:17.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:17.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:17.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc016b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:17.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,2] 2023-11-02T14:38:17.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc007cf0 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:17.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:17.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00a7f0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:17.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc007cf0 2023-11-02T14:38:17.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:17.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00a7f0 2023-11-02T14:38:17.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:17.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:17.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:17.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.461+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc00e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:17.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.662+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:17.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.662+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:17.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:17.662+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc00e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:18.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:18.063+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:18.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:18.063+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:18.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:18.063+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc00e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:18.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:18.864+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:18.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:18.864+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:18.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:18.864+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc00e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:20.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:20.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:20.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:20.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:20.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:20.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00a7f0 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:20.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc007cf0 legacy=0x7f54bc00e1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:20.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-11-02T14:38:20.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc017300 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:20.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:20.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 0x7f54bc015cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:20.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc012970 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:20.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:20.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc00c4a0 2023-11-02T14:38:20.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc017300 2023-11-02T14:38:20.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:20.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 msgr2=0x7f54bc015cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:20.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc012970 2023-11-02T14:38:20.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 0x7f54bc015cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:20.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:20.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0132d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:20.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.461+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc012260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:20.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.662+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0132d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:20.663 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.662+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 msgr2=0x7f54bc015cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:20.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.662+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 0x7f54bc015cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:20.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:20.662+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc012260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:21.064 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:21.063+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0132d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:21.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:21.063+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 msgr2=0x7f54bc015cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:21.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:21.063+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 0x7f54bc015cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:21.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:21.063+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc012260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:21.865 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:21.864+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0132d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:21.864+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 msgr2=0x7f54bc015cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:21.864+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 0x7f54bc015cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:21.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:21.864+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc012260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:23.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:23.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:23.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:23.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:23.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 msgr2=0x7f54bc015cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:23.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc012970 0x7f54bc015cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:23.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc012260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:23.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0132d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:23.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,4] 2023-11-02T14:38:23.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:23.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00fde0 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:23.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 0x7f54bc00f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:23.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc012d50 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:23.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:23.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:38:23.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:23.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc012d50 2023-11-02T14:38:23.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:23.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:23.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:23.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:23.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.462+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:23.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.663+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:23.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.663+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:23.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.663+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:23.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:23.663+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:24.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:24.064+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:24.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:24.064+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:24.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:24.064+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:24.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:24.064+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:24.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:24.865+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:24.865+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:24.865+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:24.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:24.865+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:26.463 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:26.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:26.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:26.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:26.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 msgr2=0x7f54bc00f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:26.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc012d50 0x7f54bc00f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:26.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:26.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:26.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-11-02T14:38:26.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc017300 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:26.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:26.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:26.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:26.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00fde0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:26.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc00c4a0 2023-11-02T14:38:26.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:26.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc017300 2023-11-02T14:38:26.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:26.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:38:26.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:26.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54db7fe640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:26.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:26.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.462+0000 7f54db7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:26.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.663+0000 7f54db7fe640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:26.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.663+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:26.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.663+0000 7f54db7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:26.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:26.663+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:27.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:27.064+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:27.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:27.064+0000 7f54db7fe640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:27.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:27.064+0000 7f54db7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:27.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:27.064+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:27.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:27.865+0000 7f54db7fe640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:27.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:27.865+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:27.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:27.865+0000 7f54db7fe640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:27.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:27.865+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:29.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:29.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:29.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:29.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:29.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:29.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:29.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:29.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:29.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,4] 2023-11-02T14:38:29.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:29.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00f720 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:29.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:29.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc00d1a0 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:29.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:29.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00f720 2023-11-02T14:38:29.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:29.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00d1a0 2023-11-02T14:38:29.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:29.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:29.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:29.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:29.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.462+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:29.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.663+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:29.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.663+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:29.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.663+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:29.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:29.663+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:30.065 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:30.064+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:30.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:30.064+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:30.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:30.064+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:30.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:30.064+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:30.866 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:30.865+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:30.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:30.865+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:30.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:30.865+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:30.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:30.865+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:32.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:32.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:32.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-11-02T14:38:32.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc017300 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:32.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:32.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:32.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:32.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc00d1a0 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:32.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc00c4a0 2023-11-02T14:38:32.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc017300 2023-11-02T14:38:32.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:32.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00d1a0 2023-11-02T14:38:32.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:32.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.463+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:32.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.664+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:32.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.664+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:32.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.664+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:32.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:32.664+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:33.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:33.065+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:33.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:33.065+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:33.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:33.065+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:33.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:33.065+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:33.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:33.866+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:33.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:33.866+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:33.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:33.866+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:33.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:33.866+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:35.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:35.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:35.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:35.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:35.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:35.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00d1a0 0x7f54bc00d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:35.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:35.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:35.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 10 monclient: _add_conns ranks=[2,3] 2023-11-02T14:38:35.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:35.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00fde0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:35.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:35.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00f720 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:35.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:35.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00f720 2023-11-02T14:38:35.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:35.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00fde0 2023-11-02T14:38:35.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:35.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:35.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:35.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:35.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.463+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.664+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.664+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:35.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.664+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:35.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:35.664+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:36.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:36.065+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:36.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:36.065+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:36.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:36.065+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:36.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:36.065+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:36.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:36.866+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:36.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:36.866+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:36.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:36.866+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:36.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:36.866+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:38.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:38.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:38.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:38.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:38.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:38.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:38.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 msgr2=0x7f54bc015ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:38.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00f720 0x7f54bc015ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:38.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-11-02T14:38:38.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc017300 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:38.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:38.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:38.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00fde0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:38.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:38.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc00c4a0 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:38.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc017300 2023-11-02T14:38:38.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:38.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:38.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00c4a0 2023-11-02T14:38:38.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:38.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00fde0 2023-11-02T14:38:38.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:38.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:38.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:38.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:38.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.463+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:38.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.664+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:38.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.664+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:38.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.664+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:38.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.664+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:38.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:38.664+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:39.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.065+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:39.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.065+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:39.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.065+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:39.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.065+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:39.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.065+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:39.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.866+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:39.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.866+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:39.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.866+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:39.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.866+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:39.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:39.866+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:41.464 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:41.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:41.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:41.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:41.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:41.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:41.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc014ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,0] 2023-11-02T14:38:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:41.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00d1a0 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:41.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00f720 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:41.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc00f720 2023-11-02T14:38:41.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:41.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00d1a0 2023-11-02T14:38:41.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:41.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:41.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f720 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:41.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.463+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:41.664 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.664+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.664+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f720 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:41.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:41.664+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:42.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:42.066+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f720 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:42.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:42.066+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:42.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:42.066+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:42.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:42.867+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:42.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:42.867+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f720 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:42.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:42.867+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:44.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:44.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:44.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:44.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:44.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:44.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:44.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00f720 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:44.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-11-02T14:38:44.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:44.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00fde0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:44.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc01bdc0 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:44.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:44.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:44.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc00c4a0 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:44.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc01bdc0 2023-11-02T14:38:44.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:44.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00c4a0 2023-11-02T14:38:44.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:44.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00fde0 2023-11-02T14:38:44.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:44.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:44.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc01e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:44.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:44.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:44.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.464+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:44.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.665+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc01e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:44.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.665+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:44.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.665+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:44.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.665+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:44.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:44.665+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:45.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.066+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.066+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc01e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.066+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:45.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.066+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:45.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.066+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:45.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.867+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc01e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:45.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.867+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:45.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.867+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:45.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.867+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:45.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:45.867+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:47.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:47.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:47.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:47.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:47.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:47.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:47.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 msgr2=0x7f54bc0153b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:47.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc00c4a0 0x7f54bc0153b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:47.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc01e1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:47.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,3] 2023-11-02T14:38:47.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc015ed0 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:47.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:47.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00d1a0 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:47.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc015ed0 2023-11-02T14:38:47.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:47.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00d1a0 2023-11-02T14:38:47.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:47.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:47.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:47.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.464+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:47.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.665+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:47.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.665+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:47.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:47.665+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:48.065+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:48.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:48.065+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:48.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:48.065+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:48.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:48.866+0000 7f54d3fff640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:48.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:48.866+0000 7f54d3fff640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:48.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:48.866+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:50.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:50.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:50.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:50.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 msgr2=0x7f54bc0158f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00d1a0 0x7f54bc0158f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:50.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-11-02T14:38:50.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 0x7f54bc017300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:50.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc0169e0 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:50.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:50.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00fde0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:50.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:50.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc01bdc0 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:50.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc01bdc0 2023-11-02T14:38:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc0169e0 2023-11-02T14:38:50.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:50.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00fde0 2023-11-02T14:38:50.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:50.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:50.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:50.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:50.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:50.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.463+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:50.665 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.664+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:50.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.664+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:50.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.664+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:50.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.664+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:50.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:50.664+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:51.066 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.065+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:51.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.065+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:51.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.065+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:51.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.065+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:51.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.065+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:51.867 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.866+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:51.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.866+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:51.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.866+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:51.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.866+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:51.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:51.866+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:53.465 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:53.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:53.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:53.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:53.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:53.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:53.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:53.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc0169e0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:53.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc01bdc0 legacy=0x7f54bc014230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:53.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,4] 2023-11-02T14:38:53.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:53.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 0x7f54bc014940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:53.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc015ed0 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:53.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc00c4a0 2023-11-02T14:38:53.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:53.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc015ed0 2023-11-02T14:38:53.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:53.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 msgr2=0x7f54bc014940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:53.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:53.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.464+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 0x7f54bc014940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:53.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.665+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 msgr2=0x7f54bc014940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:53.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.665+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:53.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:53.665+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 0x7f54bc014940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:54.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:54.066+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:54.066+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 msgr2=0x7f54bc014940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:54.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:54.066+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 0x7f54bc014940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:54.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:54.867+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 msgr2=0x7f54bc014940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:54.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:54.867+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:54.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:54.867+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 0x7f54bc014940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:56.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:56.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:56.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:56.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:56.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 msgr2=0x7f54bc014940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:56.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc015ed0 0x7f54bc014940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:56.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:56.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-11-02T14:38:56.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:56.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:38:56.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00fde0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:38:56.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.151:6789/0 2023-11-02T14:38:56.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc015ed0 addr v1:172.21.15.178:6789/0 2023-11-02T14:38:56.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc00c4a0 2023-11-02T14:38:56.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc015ed0 2023-11-02T14:38:56.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:56.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:56.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:38:56.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:56.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:56.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:56.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.464+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:56.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.665+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:56.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.665+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:56.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.665+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:56.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:56.665+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:57.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:57.066+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:57.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:57.066+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:57.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:57.066+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:38:57.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:57.066+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:57.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:57.867+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:38:57.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:57.867+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:38:57.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:57.867+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:38:57.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:57.867+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:38:59.466 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:38:59.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:38:59.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:38:59.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:38:59.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:59.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:38:59.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc015ed0 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:59.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:38:59.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,4] 2023-11-02T14:38:59.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 0x7f54bc013210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:59.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc017300 addr v2:172.21.15.151:3300/0 2023-11-02T14:38:59.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:38:59.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc013750 addr v2:172.21.15.178:3300/0 2023-11-02T14:38:59.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:59.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc017300 2023-11-02T14:38:59.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:38:59.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc013750 2023-11-02T14:38:59.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:38:59.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:59.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:59.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:59.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.464+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:38:59.666 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.665+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:38:59.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.665+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:38:59.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.665+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:38:59.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:38:59.665+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:00.067 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:00.066+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:00.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:00.066+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:00.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:00.066+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:00.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:00.066+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:00.868 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:00.867+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:00.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:00.867+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:00.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:00.867+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:00.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:00.867+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:02.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:39:02.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:39:02.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:39:02.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:39:02.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:02.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:02.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:02.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc017300 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:02.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-11-02T14:39:02.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.151:6789/0 2023-11-02T14:39:02.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:39:02.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:02.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc013750 addr v2:172.21.15.178:3300/0 2023-11-02T14:39:02.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:02.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00fde0 addr v2:172.21.15.151:3300/0 2023-11-02T14:39:02.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc00c4a0 2023-11-02T14:39:02.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:02.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:02.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:39:02.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:02.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc013750 2023-11-02T14:39:02.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:39:02.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:02.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:02.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:02.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.465+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:02.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.666+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:02.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.666+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:02.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.666+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:02.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.666+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:02.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:02.666+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:03.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.067+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:03.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.067+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:03.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.067+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:03.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.067+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:03.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.067+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:03.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.868+0000 7f54daffd640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:03.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.868+0000 7f54db7fe640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:03.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.868+0000 7f54db7fe640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:03.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.868+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:03.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:03.868+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:05.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:39:05.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:39:05.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:39:05.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:39:05.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:05.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013750 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:05.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:05.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 10 monclient: _add_conns ranks=[1,2] 2023-11-02T14:39:05.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc017300 addr v1:172.21.15.178:6789/0 2023-11-02T14:39:05.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 0x7f54bc013210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:05.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc015b40 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:39:05.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc017300 2023-11-02T14:39:05.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc015b40 2023-11-02T14:39:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:39:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:05.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:05.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.465+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:05.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.666+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:05.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.666+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:05.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:05.666+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:06.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:06.067+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:06.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:06.067+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:06.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:06.067+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:06.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:06.868+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:06.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:06.868+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:06.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:06.868+0000 7f54daffd640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:08.467 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:39:08.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:39:08.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:39:08.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:39:08.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 msgr2=0x7f54bc013210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:08.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc015b40 0x7f54bc013210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:08.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:08.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-11-02T14:39:08.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:08.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00fde0 addr v2:172.21.15.151:3300/0 2023-11-02T14:39:08.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:39:08.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc017300 addr v1:172.21.15.178:6789/0 2023-11-02T14:39:08.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.151:6789/0 2023-11-02T14:39:08.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc00c4a0 2023-11-02T14:39:08.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:08.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc13bf30 con 0x7f54bc017300 2023-11-02T14:39:08.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:08.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:39:08.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:08.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:39:08.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:08.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.465+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:08.667 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.666+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:08.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.666+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:08.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.666+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:08.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:08.666+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:09.068 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:09.067+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:09.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:09.067+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:09.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:09.067+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:09.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:09.067+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:09.869 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:09.868+0000 7f54db7fe640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:09.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:09.868+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:09.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:09.868+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:09.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:09.868+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:11.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:39:11.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:39:11.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:39:11.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:39:11.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:11.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:11.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc017300 legacy=0x7f54bc0169e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:11.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc0162d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:11.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 10 monclient: _add_conns ranks=[2,4] 2023-11-02T14:39:11.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 0x7f54bc013690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:11.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc011200 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:39:11.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 0x7f54bc01bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:11.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc013bd0 addr v2:172.21.15.178:3300/0 2023-11-02T14:39:11.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:11.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc013bd0 2023-11-02T14:39:11.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:11.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc011200 2023-11-02T14:39:11.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:39:11.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 msgr2=0x7f54bc013690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:11.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 0x7f54bc013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:11.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:11.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.466+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:11.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.667+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 msgr2=0x7f54bc013690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:11.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.667+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 0x7f54bc013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:11.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.667+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:11.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:11.667+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:12.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:12.068+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 msgr2=0x7f54bc013690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:12.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:12.068+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 0x7f54bc013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:12.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:12.068+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:12.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:12.068+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:12.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:12.869+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 msgr2=0x7f54bc013690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:12.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:12.869+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 0x7f54bc013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:12.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:12.869+0000 7f54d3fff640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:12.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:12.869+0000 7f54d3fff640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:14.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:39:14.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:39:14.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:39:14.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:39:14.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 msgr2=0x7f54bc013690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:14.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc011200 0x7f54bc013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:14.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 msgr2=0x7f54bc01bdc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:14.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc013bd0 0x7f54bc01bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:14.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-11-02T14:39:14.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:14.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:39:14.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00fde0 addr v2:172.21.15.151:3300/0 2023-11-02T14:39:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 0x7f54bc017300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00c4a0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:39:14.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc00d5d0 addr v1:172.21.15.178:6789/0 2023-11-02T14:39:14.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc00d5d0 2023-11-02T14:39:14.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:14.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:39:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc00c4a0 2023-11-02T14:39:14.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:39:14.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:14.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:14.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:14.477 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.466+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00d5d0 legacy=0x7f54bc00d9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:14.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.667+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:14.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.667+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:14.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.667+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:14.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.667+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:14.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:14.667+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00d5d0 legacy=0x7f54bc00d9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:15.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.068+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:15.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.068+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:15.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.068+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:15.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.068+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:15.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.068+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00d5d0 legacy=0x7f54bc00d9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:15.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.869+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:15.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.869+0000 7f54d3fff640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:15.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.869+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:15.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.869+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:15.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:15.869+0000 7f54db7fe640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00d5d0 legacy=0x7f54bc00d9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:17.468 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:39:17.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:39:17.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:39:17.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:39:17.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 msgr2=0x7f54bc017300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:17.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00c4a0 0x7f54bc017300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:17.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:17.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:17.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc00d5d0 legacy=0x7f54bc00d9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:17.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 10 monclient: _add_conns ranks=[4,0] 2023-11-02T14:39:17.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 0x7f54bc00e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:17.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc011200 addr v2:172.21.15.178:3300/0 2023-11-02T14:39:17.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc013610 addr v1:172.21.15.151:6789/0 2023-11-02T14:39:17.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc013610 2023-11-02T14:39:17.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:17.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc011200 2023-11-02T14:39:17.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:39:17.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc013610 legacy=0x7f54bc015a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:17.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.466+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 msgr2=0x7f54bc00e0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:17.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.467+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 0x7f54bc00e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:17.668 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.667+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc013610 legacy=0x7f54bc015a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:17.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.668+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 msgr2=0x7f54bc00e0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:17.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:17.668+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 0x7f54bc00e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:18.069 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:18.068+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc013610 legacy=0x7f54bc015a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:18.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:18.069+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 msgr2=0x7f54bc00e0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:18.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:18.069+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 0x7f54bc00e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:18.870 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:18.869+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc013610 legacy=0x7f54bc015a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:18.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:18.870+0000 7f54daffd640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 msgr2=0x7f54bc00e0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:18.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:18.870+0000 7f54daffd640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 0x7f54bc00e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:20.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:39:20.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:39:20.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:39:20.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:39:20.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 msgr2=0x7f54bc00e0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:20.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc011200 0x7f54bc00e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:20.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc013610 legacy=0x7f54bc015a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:20.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-11-02T14:39:20.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:20.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54db7fe640 1 -- reap_dead start 2023-11-02T14:39:20.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a con 0x7f54bc00fde0 addr v2:172.21.15.151:3300/0 2023-11-02T14:39:20.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54bc00c4a0 addr v1:172.21.15.151:6789/0 2023-11-02T14:39:20.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 0x7f54bc0172e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:20.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c con 0x7f54bc0111d0 addr v2:172.21.15.178:3300/0 2023-11-02T14:39:20.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc005480 con 0x7f54bc00c4a0 2023-11-02T14:39:20.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:20.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00fde0 2023-11-02T14:39:20.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:20.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc071990 con 0x7f54bc0111d0 2023-11-02T14:39:20.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:39:20.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:20.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:20.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:20.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 msgr2=0x7f54bc0172e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:20.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.467+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 0x7f54bc0172e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:20.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.668+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:20.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.668+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:20.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.668+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:20.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.668+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 msgr2=0x7f54bc0172e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:20.671 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:20.668+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 0x7f54bc0172e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:21.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.069+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:21.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.069+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:21.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.069+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:21.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.069+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 msgr2=0x7f54bc0172e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:21.072 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.069+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 0x7f54bc0172e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.870+0000 7f54d3fff640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.151:6789/0 2023-11-02T14:39:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.870+0000 7f54daffd640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3300/0 2023-11-02T14:39:21.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.870+0000 7f54daffd640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:21.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.870+0000 7f54db7fe640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 msgr2=0x7f54bc0172e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:3300/0 2023-11-02T14:39:21.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:21.870+0000 7f54db7fe640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 0x7f54bc0172e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:23.469 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 10 monclient(hunting): tick 2023-11-02T14:39:23.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 10 monclient(hunting): _check_auth_tickets 2023-11-02T14:39:23.470 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 1 monclient(hunting): continuing hunt 2023-11-02T14:39:23.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 10 monclient(hunting): _reopen_session rank -1 2023-11-02T14:39:23.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 1 -- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 msgr2=0x7f54bc0172e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:23.471 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54bc0111d0 0x7f54bc0172e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:23.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 1 -- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 msgr2=0x7f54bc00b740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:23.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f54bc00fde0 0x7f54bc00b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-02T14:39:23.472 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 1 -- >> v1:172.21.15.151:6789/0 conn(0x7f54bc00c4a0 legacy=0x7f54bc00d5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-02T14:39:23.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 10 monclient: _add_conns ranks=[2,1] 2023-11-02T14:39:23.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00dce0 0x7f54bc00e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-02T14:39:23.473 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-b con 0x7f54bc00dce0 addr [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] 2023-11-02T14:39:23.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54bc013610 addr v1:172.21.15.178:6789/0 2023-11-02T14:39:23.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0aeae0 con 0x7f54bc013610 2023-11-02T14:39:23.474 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 10 monclient(hunting): start opening mon connection 2023-11-02T14:39:23.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0ae840 con 0x7f54bc00dce0 2023-11-02T14:39:23.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d97fa640 10 monclient(hunting): _renew_subs 2023-11-02T14:39:23.475 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00dce0 msgr2=0x7f54bc00e140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:23.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00dce0 0x7f54bc00e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-02T14:39:23.476 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.467+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc013610 legacy=0x7f54bc015a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:23.669 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.668+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00dce0 msgr2=0x7f54bc00e140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:23.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.668+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00dce0 0x7f54bc00e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-02T14:39:23.670 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:23.668+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc013610 legacy=0x7f54bc015a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:24.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:24.069+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00dce0 msgr2=0x7f54bc00e140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:24.070 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:24.069+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00dce0 0x7f54bc00e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-02T14:39:24.071 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:24.069+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc013610 legacy=0x7f54bc015a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:24.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:24.870+0000 7f54daffd640 1 -- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00dce0 msgr2=0x7f54bc00e140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.151:3301/0 2023-11-02T14:39:24.872 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:24.870+0000 7f54daffd640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f54bc00dce0 0x7f54bc00e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-02T14:39:24.873 INFO:teuthology.orchestra.run.smithi151.stderr:2023-11-02T14:39:24.870+0000 7f54d3fff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7f54bc013610 legacy=0x7f54bc015a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2023-11-02T14:39:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-02T14:39:26.389 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2745, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2577, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2343, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-02T14:39:26.401 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-02T14:39:26.401 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-02T14:39:26.402 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-02T14:39:26.402 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.405 INFO:tasks.ceph.osd.0:Stopped 2023-11-02T14:39:26.406 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-02T14:39:26.406 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.407 INFO:tasks.ceph.osd.1:Stopped 2023-11-02T14:39:26.407 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-02T14:39:26.408 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-11-02T14:39:26.409 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2745, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2577, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2343, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2023-11-02T14:39:26.419 INFO:tasks.ceph.osd.2:Stopped 2023-11-02T14:39:26.419 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-02T14:39:26.420 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.420 INFO:tasks.ceph.osd.3:Stopped 2023-11-02T14:39:26.421 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-02T14:39:26.421 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.422 INFO:tasks.ceph.osd.4:Stopped 2023-11-02T14:39:26.422 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-02T14:39:26.423 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.423 INFO:tasks.ceph.osd.5:Stopped 2023-11-02T14:39:26.424 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-02T14:39:26.424 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.425 INFO:tasks.ceph.osd.6:Stopped 2023-11-02T14:39:26.425 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-02T14:39:26.426 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.426 INFO:tasks.ceph.osd.7:Stopped 2023-11-02T14:39:26.427 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-02T14:39:26.427 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-02T14:39:26.428 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T14:39:26.429 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2745, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2577, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2343, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y' 2023-11-02T14:39:26.439 INFO:tasks.ceph.mgr.y:Stopped 2023-11-02T14:39:26.440 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-02T14:39:26.440 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.441 INFO:tasks.ceph.mgr.x:Stopped 2023-11-02T14:39:26.442 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-02T14:39:26.442 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-02T14:39:26.443 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-11-02T14:39:26.444 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2745, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2577, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2343, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-11-02T14:39:26.456 INFO:tasks.ceph.mon.a:Stopped 2023-11-02T14:39:26.457 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-02T14:39:26.457 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-11-02T14:39:26.458 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2745, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2577, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2343, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-11-02T14:39:26.468 INFO:tasks.ceph.mon.c:Stopped 2023-11-02T14:39:26.469 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-02T14:39:26.469 INFO:teuthology.orchestra.run:waiting for 300 2023-11-02T14:39:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-11-02T14:39:26.470 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2745, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2577, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2343, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-11-02T14:39:26.481 INFO:tasks.ceph.mon.b:Stopped 2023-11-02T14:39:26.482 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-02T14:39:26.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-11-02T14:39:26.527 INFO:teuthology.orchestra.run.smithi151.stdout:2023-11-02T08:40:50.606118+0000 osd.7 (osd.7) 4 : cluster [WRN] osd.7 ep: 155 scrubber::ReplicaReservations pg[11.5]: timeout on replica reservations (since 2023-11-02 08:40:45) 2023-11-02T14:39:26.528 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-11-02T14:39:26.528 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T14:39:26.529 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-02T14:39:26.662 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T14:39:26.663 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-02T14:39:26.745 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T14:39:26.746 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-02T14:39:26.810 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T14:39:26.811 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-02T14:39:26.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T14:39:26.868 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-02T14:39:27.020 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T14:39:27.021 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-02T14:39:27.090 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T14:39:27.091 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-02T14:39:27.160 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T14:39:27.161 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-02T14:39:27.252 INFO:tasks.ceph:Archiving mon data... 2023-11-02T14:39:27.253 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/data/mon.a.tgz 2023-11-02T14:39:27.254 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2023-11-02T14:39:27.269 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.Yldcp2Edxh 2023-11-02T14:39:27.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Yldcp2Edxh 2023-11-02T14:39:27.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.Yldcp2Edxh 2023-11-02T14:39:27.524 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.Yldcp2Edxh is 463KB 2023-11-02T14:39:27.622 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.Yldcp2Edxh 2023-11-02T14:39:27.638 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/data/mon.c.tgz 2023-11-02T14:39:27.639 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2023-11-02T14:39:27.692 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.h6FIWdVucO 2023-11-02T14:39:27.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.h6FIWdVucO 2023-11-02T14:39:27.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.h6FIWdVucO 2023-11-02T14:39:27.931 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.h6FIWdVucO is 464KB 2023-11-02T14:39:28.028 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.h6FIWdVucO 2023-11-02T14:39:28.042 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/data/mon.b.tgz 2023-11-02T14:39:28.043 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2023-11-02T14:39:28.058 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.7gLXy11AVX 2023-11-02T14:39:28.058 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.7gLXy11AVX 2023-11-02T14:39:28.227 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.7gLXy11AVX 2023-11-02T14:39:28.313 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.7gLXy11AVX is 464KB 2023-11-02T14:39:28.410 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.7gLXy11AVX 2023-11-02T14:39:28.425 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-02T14:39:28.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-02T14:39:28.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-02T14:39:28.490 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-02T14:39:28.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-02T14:39:28.496 DEBUG:teuthology.orchestra.run.smithi178:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-02T14:39:28.535 INFO:teuthology.orchestra.run.smithi151.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2023-11-02T14:39:28.536 INFO:teuthology.orchestra.run.smithi151.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2023-11-02T14:39:28.536 INFO:teuthology.orchestra.run.smithi151.stdout:/var/log/ceph/valgrind/osd.2.log: Leak_PossiblyLost 2023-11-02T14:39:28.537 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2023-11-02T14:39:28.538 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2023-11-02T14:39:28.542 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2023-11-02T14:39:28.542 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2023-11-02T14:39:28.547 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.2.log kind Leak_PossiblyLost 2023-11-02T14:39:28.548 ERROR:tasks.ceph:saw valgrind issue Leak_PossiblyLost in /var/log/ceph/valgrind/osd.2.log 2023-11-02T14:39:28.570 INFO:teuthology.orchestra.run.smithi178.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2023-11-02T14:39:28.571 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2023-11-02T14:39:28.571 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2023-11-02T14:39:28.575 INFO:tasks.ceph:Archiving crash dumps... 2023-11-02T14:39:28.578 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/remote/smithi151/crash 2023-11-02T14:39:28.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-02T14:39:28.657 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/remote/smithi178/crash 2023-11-02T14:39:28.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-02T14:39:28.685 INFO:tasks.ceph:Compressing logs... 2023-11-02T14:39:28.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-02T14:39:28.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-02T14:39:50.695 INFO:tasks.ceph:Archiving logs... 2023-11-02T14:39:50.696 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/remote/smithi151/log 2023-11-02T14:39:50.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-02T14:39:53.421 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/remote/smithi178/log 2023-11-02T14:39:53.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-02T14:39:55.529 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 322, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2745, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2577, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2343, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-11-02T14:39:55.546 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-02T14:39:55.556 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 322, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2745, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2577, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2343, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-11-02T14:39:55.562 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-02T14:39:55.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-02T14:39:55.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-02T14:39:55.658 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-02T14:39:55.668 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-02T14:39:55.668 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T14:39:55.671 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T14:39:55.685 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2023-11-02T14:39:55.688 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2023-11-02T14:39:55.689 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-02T14:39:55.690 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2023-11-02T14:39:55.690 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 29 +575us[ +575us] +/- 85ms 2023-11-02T14:39:55.691 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv02.front.sepia.ceph.com 3 7 377 98 -548us[ -540us] +/- 62ms 2023-11-02T14:39:55.691 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 164 +267us[ +273us] +/- 87ms 2023-11-02T14:39:55.691 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T14:39:55.692 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-02T14:39:55.693 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2023-11-02T14:39:55.693 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 9 +597us[ +597us] +/- 85ms 2023-11-02T14:39:55.694 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -555us[ -556us] +/- 62ms 2023-11-02T14:39:55.694 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 203 +290us[ +260us] +/- 87ms 2023-11-02T14:39:55.694 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T14:39:55.695 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-02T14:39:55.704 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-02T14:39:55.706 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-02T14:39:55.748 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-02T14:39:55.847 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-02T14:39:55.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-02T14:39:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T14:39:55.981 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-02T14:39:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T14:39:56.016 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-02T14:39:56.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-02T14:39:56.045 INFO:teuthology.task.internal:Duration was 22806.223714 seconds 2023-11-02T14:39:56.046 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-02T14:39:56.078 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-02T14:39:56.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-02T14:39:56.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-02T14:39:56.123 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-02T14:39:56.127 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-02T14:39:56.476 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-02T14:39:56.477 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2023-11-02T14:39:56.477 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-02T14:39:56.536 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2023-11-02T14:39:56.536 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-02T14:39:56.571 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-02T14:39:56.572 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-02T14:39:56.578 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-02T14:39:56.640 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-02T14:39:56.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-02T14:39:56.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-02T14:39:56.787 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-02T14:39:56.798 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-02T14:39:56.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-02T14:39:56.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-02T14:39:56.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-02T14:39:56.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-02T14:39:56.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-02T14:39:56.891 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2023-11-02T14:39:56.918 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2023-11-02T14:39:56.967 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-02T14:39:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T14:39:56.982 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-02T14:39:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T14:39:57.021 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-02T14:39:57.032 INFO:teuthology.task.internal:Transferring archived files... 2023-11-02T14:39:57.033 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/remote/smithi151 2023-11-02T14:39:57.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-02T14:39:57.071 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888/remote/smithi178 2023-11-02T14:39:57.071 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-02T14:39:57.106 INFO:teuthology.task.internal:Removing archive directory... 2023-11-02T14:39:57.107 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-02T14:39:57.110 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-02T14:39:57.155 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-02T14:39:57.167 INFO:teuthology.task.internal:Not uploading archives. 2023-11-02T14:39:57.167 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-02T14:39:57.188 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-02T14:39:57.188 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-02T14:39:57.191 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-02T14:39:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 2 14:39 /home/ubuntu/cephtest 2023-11-02T14:39:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: 277624 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 2 08:36 /home/ubuntu/cephtest/mnt.0 2023-11-02T14:39:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: 277625 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 2 08:37 /home/ubuntu/cephtest/mnt.0/client.0 2023-11-02T14:39:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: 291383 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 2 08:37 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T14:39:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: 291404 16 -rw-r--r-- 1 ubuntu ubuntu 16384 Nov 2 09:01 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_tier_pp.log 2023-11-02T14:39:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: 291411 4 -rw-r--r-- 1 ubuntu ubuntu 1278 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat_pp.log 2023-11-02T14:39:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: 291389 4 -rw-r--r-- 1 ubuntu ubuntu 1874 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock.log 2023-11-02T14:39:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: 291428 4 -rw-r--r-- 1 ubuntu ubuntu 1705 Nov 2 08:38 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_read_operations.log 2023-11-02T14:39:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: 291430 8 -rw-r--r-- 1 ubuntu ubuntu 5612 Nov 2 08:42 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_list_parallel.log 2023-11-02T14:39:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: 291397 8 -rw-r--r-- 1 ubuntu ubuntu 4273 Nov 2 08:42 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc_pp.log 2023-11-02T14:39:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: 291384 4 -rw-r--r-- 1 ubuntu ubuntu 4043 Nov 2 08:44 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio.log 2023-11-02T14:39:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: 291394 4 -rw-r--r-- 1 ubuntu ubuntu 2312 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc.log 2023-11-02T14:39:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: 291434 4 -rw-r--r-- 1 ubuntu ubuntu 423 Nov 2 08:38 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cls_remote_reads.log 2023-11-02T14:39:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: 291418 4 -rw-r--r-- 1 ubuntu ubuntu 592 Nov 2 08:38 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd_pp.log 2023-11-02T14:39:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: 291417 16 -rw-r--r-- 1 ubuntu ubuntu 15797 Nov 2 08:38 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd.log 2023-11-02T14:39:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: 291415 4 -rw-r--r-- 1 ubuntu ubuntu 2716 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify.log 2023-11-02T14:39:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: 291406 4 -rw-r--r-- 1 ubuntu ubuntu 2069 Nov 2 08:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots.log 2023-11-02T14:39:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: 291387 104 -rw-r--r-- 1 ubuntu ubuntu 104368 Nov 2 08:37 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_asio.log 2023-11-02T14:39:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: 291410 4 -rw-r--r-- 1 ubuntu ubuntu 1173 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat.log 2023-11-02T14:39:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: 291431 4 -rw-r--r-- 1 ubuntu ubuntu 1311 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_delete_pools_parallel.log 2023-11-02T14:39:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: 291425 4 -rw-r--r-- 1 ubuntu ubuntu 713 Nov 2 08:37 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service_pp.log 2023-11-02T14:39:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: 291427 4 -rw-r--r-- 1 ubuntu ubuntu 1203 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_write_operations.log 2023-11-02T14:39:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: 291421 4 -rw-r--r-- 1 ubuntu ubuntu 2076 Nov 2 08:37 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service.log 2023-11-02T14:39:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: 291385 8 -rw-r--r-- 1 ubuntu ubuntu 5869 Nov 2 08:47 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio_pp.log 2023-11-02T14:39:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: 291388 4 -rw-r--r-- 1 ubuntu ubuntu 3899 Nov 2 08:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io_pp.log 2023-11-02T14:39:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: 291435 4 -rw-r--r-- 1 ubuntu ubuntu 1040 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_open_pools_parallel.log 2023-11-02T14:39:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: 291413 8 -rw-r--r-- 1 ubuntu ubuntu 4147 Nov 2 08:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify_pp.log 2023-11-02T14:39:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: 291395 4 -rw-r--r-- 1 ubuntu ubuntu 2022 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock_pp.log 2023-11-02T14:39:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: 291403 4 -rw-r--r-- 1 ubuntu ubuntu 1244 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_pool.log 2023-11-02T14:39:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: 291390 8 -rw-r--r-- 1 ubuntu ubuntu 6897 Nov 2 08:43 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_list.log 2023-11-02T14:39:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: 291408 4 -rw-r--r-- 1 ubuntu ubuntu 3374 Nov 2 08:42 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots_pp.log 2023-11-02T14:39:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: 291386 4 -rw-r--r-- 1 ubuntu ubuntu 2576 Nov 2 08:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io.log 2023-11-02T14:39:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: 291463 827472 -rw------- 1 root root 847798272 Nov 2 09:05 /home/ubuntu/cephtest/vgcore.86100 2023-11-02T14:39:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T14:39:57.215 INFO:teuthology.orchestra.run.smithi178.stdout: 265243 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 2 14:39 /home/ubuntu/cephtest 2023-11-02T14:39:57.216 INFO:teuthology.orchestra.run.smithi151.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-02T14:39:57.216 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 322, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2745, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2577, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 2343, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi151 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-02T14:39:57.234 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-02T14:39:57.244 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-02T14:39:57.264 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2023-11-02T14:39:57.264 INFO:teuthology.nuke:Checking targets against current locks 2023-11-02T14:39:57.297 DEBUG:teuthology.nuke:shortname: smithi151 2023-11-02T14:39:57.297 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-02T14:39:57.322 DEBUG:teuthology.nuke:shortname: smithi178 2023-11-02T14:39:57.322 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-02T14:39:57.326 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 08:10:14.553469', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKB+YlhA8QRvwhofhT9njZNwkD80E3wr4ljY0MpgRWCrpgX8xVCBUrUlMl/xOz6do5PWtzhVKyqQYgel4+hwS00='} 2023-11-02T14:39:57.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443888', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 08:10:14.555848', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWyM0BTTlZcSU06MdrOc/wX6OC7tSoQIDcXxB4KmJQ+YsjOrG0UckB44ObGExInMq2G8EWSWXFdPBEGaCx14rM='} 2023-11-02T14:39:57.377 INFO:teuthology.orchestra.console.smithi151:Power off 2023-11-02T14:39:57.379 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-02T14:39:57.404 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2023-11-02T14:39:57.404 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T14:39:57.419 INFO:teuthology.orchestra.console.smithi178:Power off 2023-11-02T14:39:57.420 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-02T14:39:57.430 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2023-11-02T14:39:57.442 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2023-11-02T14:39:57.442 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T14:39:57.466 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2023-11-02T14:40:01.431 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T14:40:01.468 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T14:40:01.544 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2023-11-02T14:40:01.580 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2023-11-02T14:40:05.546 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T14:40:05.582 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T14:40:05.660 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2023-11-02T14:40:05.693 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2023-11-02T14:40:05.794 INFO:teuthology.orchestra.console.smithi178:Power off completed 2023-11-02T14:40:09.664 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T14:40:09.776 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2023-11-02T14:40:09.877 INFO:teuthology.orchestra.console.smithi151:Power off completed 2023-11-02T14:40:09.983 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} duration: 22806.2237136364 failure_reason: 'Command failed (workunit test rados/test.sh) on smithi151 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=438a2aec82e9401d94b9f9a99c0d8599 status: fail success: false 2023-11-02T14:40:09.984 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-02T14:40:10.070 INFO:teuthology.run:FAIL